2024-03-26T03:56:00.368 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-26T03:56:00.369 DEBUG:teuthology.run:Teuthology command: teuthology --name teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi --owner scheduled_teuthology@teuthology --archive /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621980 --verbose --description orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_api_tests} -- /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621980/orig.config.yaml 2024-03-26T03:56:00.398 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-26T03:56:00.472 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621980 branch: squid description: orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_api_tests} email: ceph-qa@ceph.com first_in_suite: false job_id: '7621980' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: client: debug ms: 1 global: mon election default strategy: 1 ms bind msgr2: false ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon warn on pool no app: false osd: debug ms: 1 debug osd: 20 osd class default list: '*' osd class load list: '*' osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - reached quota - but it is still running - overall HEALTH_ - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(PG_DEGRADED\) mon_bind_msgr2: false sha1: 5e70124b46c2a671efcaae4690cd9ea72670710a ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: cephadm-package install: ceph: flavor: default sha1: 5e70124b46c2a671efcaae4690cd9ea72670710a extra_packages: - cephadm selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: squid sha1: 9f04c91b77e4619963d41730d8cf756e3a6152de owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 1822 sha1: 5e70124b46c2a671efcaae4690cd9ea72670710a sleep_before_teardown: 0 subset: 53/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_9f04c91b77e4619963d41730d8cf756e3a6152de/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 9f04c91b77e4619963d41730d8cf756e3a6152de targets: smithi016.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIpwUlkulJHdCP0MC8OTMn5p3Jfm/ZK9sOwgeFKr+GbAa0csykYqsEvQ62XJKhPH3NmE9+u/ddSrluG4s0e/CKk= smithi183.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBsdLSX8oqYf+V6JXV98ROd9t+2tjF/cUqQj4n2AKMPqehYqqYSAE3gzESTIJSGEATACBb8TkITN6BRY7PvODW8= tasks: - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - workunit: clients: client.0: - rados/test.sh - rados/test_pool_quota.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_21:08:11 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-26T03:56:00.472 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_9f04c91b77e4619963d41730d8cf756e3a6152de/qa; will attempt to use it 2024-03-26T03:56:00.473 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_9f04c91b77e4619963d41730d8cf756e3a6152de/qa/tasks 2024-03-26T03:56:00.473 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-26T03:56:00.475 INFO:teuthology.task.internal:Checking packages... 2024-03-26T03:56:00.498 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5e70124b46c2a671efcaae4690cd9ea72670710a' 2024-03-26T03:56:00.498 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-26T03:56:00.498 INFO:teuthology.packaging:ref: None 2024-03-26T03:56:00.499 INFO:teuthology.packaging:tag: None 2024-03-26T03:56:00.499 INFO:teuthology.packaging:branch: squid 2024-03-26T03:56:00.499 INFO:teuthology.packaging:sha1: 5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T03:56:00.499 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-26T03:56:00.651 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1863.g9f04c91b 2024-03-26T03:56:00.653 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-26T03:56:00.659 INFO:teuthology.task.internal:no buildpackages task found 2024-03-26T03:56:00.659 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-26T03:56:00.666 INFO:teuthology.task.internal:Saving configuration 2024-03-26T03:56:00.677 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-26T03:56:00.683 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T03:56:00.707 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi016.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621980', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 03:51:21.281574', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIpwUlkulJHdCP0MC8OTMn5p3Jfm/ZK9sOwgeFKr+GbAa0csykYqsEvQ62XJKhPH3NmE9+u/ddSrluG4s0e/CKk='} 2024-03-26T03:56:00.730 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi183.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621980', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 03:51:21.283180', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBsdLSX8oqYf+V6JXV98ROd9t+2tjF/cUqQj4n2AKMPqehYqqYSAE3gzESTIJSGEATACBb8TkITN6BRY7PvODW8='} 2024-03-26T03:56:00.730 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-26T03:56:00.736 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-26T03:56:00.736 INFO:teuthology.task.internal:roles: ubuntu@smithi183.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-03-26T03:56:00.736 INFO:teuthology.run_tasks:Running task console_log... 2024-03-26T03:56:00.804 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5aaf0ecf70>, signals=[15]) 2024-03-26T03:56:00.805 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-26T03:56:00.811 INFO:teuthology.task.internal:Opening connections... 2024-03-26T03:56:00.811 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi016.front.sepia.ceph.com 2024-03-26T03:56:00.813 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T03:56:00.916 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi183.front.sepia.ceph.com 2024-03-26T03:56:00.917 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T03:56:01.003 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-26T03:56:01.010 DEBUG:teuthology.orchestra.run.smithi016:> uname -m 2024-03-26T03:56:01.028 INFO:teuthology.orchestra.run.smithi016.stdout:x86_64 2024-03-26T03:56:01.028 DEBUG:teuthology.orchestra.run.smithi016:> cat /etc/os-release 2024-03-26T03:56:01.085 INFO:teuthology.orchestra.run.smithi016.stdout:NAME="CentOS Stream" 2024-03-26T03:56:01.085 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION="9" 2024-03-26T03:56:01.085 INFO:teuthology.orchestra.run.smithi016.stdout:ID="centos" 2024-03-26T03:56:01.085 INFO:teuthology.orchestra.run.smithi016.stdout:ID_LIKE="rhel fedora" 2024-03-26T03:56:01.086 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION_ID="9" 2024-03-26T03:56:01.086 INFO:teuthology.orchestra.run.smithi016.stdout:PLATFORM_ID="platform:el9" 2024-03-26T03:56:01.086 INFO:teuthology.orchestra.run.smithi016.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-26T03:56:01.086 INFO:teuthology.orchestra.run.smithi016.stdout:ANSI_COLOR="0;31" 2024-03-26T03:56:01.086 INFO:teuthology.orchestra.run.smithi016.stdout:LOGO="fedora-logo-icon" 2024-03-26T03:56:01.086 INFO:teuthology.orchestra.run.smithi016.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-26T03:56:01.086 INFO:teuthology.orchestra.run.smithi016.stdout:HOME_URL="https://centos.org/" 2024-03-26T03:56:01.086 INFO:teuthology.orchestra.run.smithi016.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-26T03:56:01.086 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-26T03:56:01.086 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-26T03:56:01.087 INFO:teuthology.lock.ops:Updating smithi016.front.sepia.ceph.com on lock server 2024-03-26T03:56:01.110 DEBUG:teuthology.orchestra.run.smithi183:> uname -m 2024-03-26T03:56:01.128 INFO:teuthology.orchestra.run.smithi183.stdout:x86_64 2024-03-26T03:56:01.128 DEBUG:teuthology.orchestra.run.smithi183:> cat /etc/os-release 2024-03-26T03:56:01.185 INFO:teuthology.orchestra.run.smithi183.stdout:NAME="CentOS Stream" 2024-03-26T03:56:01.185 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION="9" 2024-03-26T03:56:01.185 INFO:teuthology.orchestra.run.smithi183.stdout:ID="centos" 2024-03-26T03:56:01.185 INFO:teuthology.orchestra.run.smithi183.stdout:ID_LIKE="rhel fedora" 2024-03-26T03:56:01.185 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION_ID="9" 2024-03-26T03:56:01.185 INFO:teuthology.orchestra.run.smithi183.stdout:PLATFORM_ID="platform:el9" 2024-03-26T03:56:01.185 INFO:teuthology.orchestra.run.smithi183.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-26T03:56:01.185 INFO:teuthology.orchestra.run.smithi183.stdout:ANSI_COLOR="0;31" 2024-03-26T03:56:01.185 INFO:teuthology.orchestra.run.smithi183.stdout:LOGO="fedora-logo-icon" 2024-03-26T03:56:01.185 INFO:teuthology.orchestra.run.smithi183.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-26T03:56:01.185 INFO:teuthology.orchestra.run.smithi183.stdout:HOME_URL="https://centos.org/" 2024-03-26T03:56:01.186 INFO:teuthology.orchestra.run.smithi183.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-26T03:56:01.186 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-26T03:56:01.186 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-26T03:56:01.186 INFO:teuthology.lock.ops:Updating smithi183.front.sepia.ceph.com on lock server 2024-03-26T03:56:01.201 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-26T03:56:01.210 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-26T03:56:01.215 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-26T03:56:01.216 DEBUG:teuthology.orchestra.run.smithi016:> test '!' -e /home/ubuntu/cephtest 2024-03-26T03:56:01.218 DEBUG:teuthology.orchestra.run.smithi183:> test '!' -e /home/ubuntu/cephtest 2024-03-26T03:56:01.241 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-26T03:56:01.247 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-26T03:56:01.247 DEBUG:teuthology.orchestra.run.smithi016:> test -z $(ls -A /var/lib/ceph) 2024-03-26T03:56:01.275 DEBUG:teuthology.orchestra.run.smithi183:> test -z $(ls -A /var/lib/ceph) 2024-03-26T03:56:01.301 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-26T03:56:01.354 INFO:teuthology.run_tasks:Running task kernel... 2024-03-26T03:56:01.369 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-26T03:56:01.369 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-26T03:56:01.369 DEBUG:teuthology.orchestra.run.smithi016:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-26T03:56:01.370 DEBUG:teuthology.orchestra.run.smithi183:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-26T03:56:01.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:01.385 DEBUG:teuthology.orchestra.run.smithi183:> uname -r 2024-03-26T03:56:01.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:56:01.387 DEBUG:teuthology.orchestra.run.smithi016:> uname -r 2024-03-26T03:56:01.442 INFO:teuthology.orchestra.run.smithi183.stdout:5.14.0-361.el9.x86_64 2024-03-26T03:56:01.442 INFO:teuthology.task.kernel:Running kernel on smithi183: 5.14.0-361.el9.x86_64 2024-03-26T03:56:01.442 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum install -y kernel 2024-03-26T03:56:01.444 INFO:teuthology.orchestra.run.smithi016.stdout:5.14.0-361.el9.x86_64 2024-03-26T03:56:01.445 INFO:teuthology.task.kernel:Running kernel on smithi016: 5.14.0-361.el9.x86_64 2024-03-26T03:56:01.445 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum install -y kernel 2024-03-26T03:56:03.415 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-26T03:56:03.565 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-26T03:56:05.015 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.0 MB 00:00 2024-03-26T03:56:05.141 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - BaseOS 3.3 MB/s | 2.0 MB 00:00 2024-03-26T03:56:06.317 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - BaseOS 9.6 MB/s | 8.0 MB 00:00 2024-03-26T03:56:07.081 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.0 MB 00:01 2024-03-26T03:56:23.033 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:16 2024-03-26T03:56:25.172 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - AppStream 1.1 MB/s | 19 MB 00:16 2024-03-26T03:56:31.081 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - CRB 1.3 MB/s | 6.1 MB 00:04 2024-03-26T03:56:31.392 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - CRB 2.4 MB/s | 6.1 MB 00:02 2024-03-26T03:56:32.548 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - Extras packages 48 kB/s | 16 kB 00:00 2024-03-26T03:56:33.257 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - Extras packages 23 kB/s | 16 kB 00:00 2024-03-26T03:56:34.683 INFO:teuthology.orchestra.run.smithi016.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-26T03:56:35.616 INFO:teuthology.orchestra.run.smithi183.stdout:Extra Packages for Enterprise Linux 7.0 MB/s | 21 MB 00:02 2024-03-26T03:56:39.750 INFO:teuthology.orchestra.run.smithi016.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-26T03:56:40.741 INFO:teuthology.orchestra.run.smithi183.stdout:lab-extras 6.1 kB/s | 1.7 kB 00:00 2024-03-26T03:56:42.016 INFO:teuthology.orchestra.run.smithi016.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-26T03:56:42.016 INFO:teuthology.orchestra.run.smithi016.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-26T03:56:42.104 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-26T03:56:42.114 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-26T03:56:42.114 INFO:teuthology.orchestra.run.smithi016.stdout: Package Architecture Version Repository Size 2024-03-26T03:56:42.114 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-26T03:56:42.114 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-26T03:56:42.114 INFO:teuthology.orchestra.run.smithi016.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-26T03:56:42.114 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-03-26T03:56:42.114 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-26T03:56:42.114 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-26T03:56:42.115 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-26T03:56:42.115 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T03:56:42.115 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-26T03:56:42.115 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-26T03:56:42.115 INFO:teuthology.orchestra.run.smithi016.stdout:Install 4 Packages 2024-03-26T03:56:42.115 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T03:56:42.115 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 76 M 2024-03-26T03:56:42.115 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 123 M 2024-03-26T03:56:42.116 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-26T03:56:42.243 INFO:teuthology.orchestra.run.smithi016.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 957 kB/s | 66 kB 00:00 2024-03-26T03:56:42.906 INFO:teuthology.orchestra.run.smithi183.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-26T03:56:42.906 INFO:teuthology.orchestra.run.smithi183.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-26T03:56:42.999 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-26T03:56:43.009 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-26T03:56:43.009 INFO:teuthology.orchestra.run.smithi183.stdout: Package Architecture Version Repository Size 2024-03-26T03:56:43.009 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-26T03:56:43.009 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-26T03:56:43.009 INFO:teuthology.orchestra.run.smithi183.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-26T03:56:43.009 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-26T03:56:43.010 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-26T03:56:43.010 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-26T03:56:43.010 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-26T03:56:43.010 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T03:56:43.010 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-26T03:56:43.010 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-26T03:56:43.010 INFO:teuthology.orchestra.run.smithi183.stdout:Install 4 Packages 2024-03-26T03:56:43.010 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T03:56:43.011 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 76 M 2024-03-26T03:56:43.011 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 123 M 2024-03-26T03:56:43.011 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-26T03:56:43.060 INFO:teuthology.orchestra.run.smithi016.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 17 MB/s | 15 MB 00:00 2024-03-26T03:56:43.143 INFO:teuthology.orchestra.run.smithi183.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T03:56:43.143 INFO:teuthology.orchestra.run.smithi183.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T03:56:43.144 INFO:teuthology.orchestra.run.smithi183.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T03:56:43.287 INFO:teuthology.orchestra.run.smithi183.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T03:56:43.295 INFO:teuthology.orchestra.run.smithi183.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T03:56:43.320 INFO:teuthology.orchestra.run.smithi183.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T03:56:43.634 INFO:teuthology.orchestra.run.smithi183.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 113 kB/s | 66 kB 00:00 2024-03-26T03:56:43.665 INFO:teuthology.orchestra.run.smithi183.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T03:56:43.697 INFO:teuthology.orchestra.run.smithi183.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-26T03:56:43.777 INFO:teuthology.orchestra.run.smithi016.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 21 MB/s | 33 MB 00:01 2024-03-26T03:56:44.036 INFO:teuthology.orchestra.run.smithi016.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 15 MB/s | 27 MB 00:01 2024-03-26T03:56:44.036 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-26T03:56:44.036 INFO:teuthology.orchestra.run.smithi016.stdout:Total 40 MB/s | 76 MB 00:01 2024-03-26T03:56:44.733 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-26T03:56:45.177 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-26T03:56:45.177 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-26T03:56:46.218 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-26T03:56:46.218 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-26T03:56:48.905 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-26T03:56:49.552 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-26T03:56:49.644 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T03:56:52.523 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T03:56:52.655 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T03:56:55.824 INFO:teuthology.orchestra.run.smithi183.stdout:(2/4): kernel-modules-core-5.14.0-432.el9.x86_6 2.3 MB/s | 27 MB 00:12 2024-03-26T03:56:56.165 INFO:teuthology.orchestra.run.smithi183.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 2.6 MB/s | 33 MB 00:13 2024-03-26T03:56:56.898 INFO:teuthology.orchestra.run.smithi183.stdout:(4/4): kernel-core-5.14.0-432.el9.x86_64.rpm 1.1 MB/s | 15 MB 00:13 2024-03-26T03:56:56.902 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-26T03:56:56.902 INFO:teuthology.orchestra.run.smithi183.stdout:Total 5.5 MB/s | 76 MB 00:13 2024-03-26T03:56:57.566 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-26T03:56:57.731 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T03:56:57.774 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T03:56:57.940 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-26T03:56:57.940 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-26T03:56:58.962 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-26T03:56:58.962 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-26T03:57:01.426 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-26T03:57:02.064 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-26T03:57:02.157 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T03:57:02.874 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T03:57:04.889 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T03:57:05.018 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T03:57:09.904 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T03:57:09.943 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T03:57:14.705 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T03:57:33.922 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T03:57:33.922 INFO:teuthology.orchestra.run.smithi016.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-26T03:57:33.923 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T03:57:33.967 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-26T03:57:34.355 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T03:57:34.356 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-26T03:57:34.356 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T03:57:34.356 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T03:57:34.785 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T03:57:34.785 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T03:57:34.785 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-26T03:57:34.786 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-26T03:57:34.786 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-26T03:57:34.786 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-26T03:57:34.786 INFO:teuthology.orchestra.run.smithi016.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-26T03:57:34.786 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T03:57:34.786 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-26T03:57:35.108 DEBUG:teuthology.orchestra.run.smithi016:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T03:57:35.203 INFO:teuthology.orchestra.run.smithi016.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T03:57:35.203 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T03:57:35.203 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-26T03:57:35.204 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-26T03:57:35.204 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-26T03:57:35.204 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-26T03:57:35.204 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi016.front.sepia.ceph.com, path=None, version=distro) 2024-03-26T03:57:35.204 DEBUG:teuthology.orchestra.run.smithi016:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T03:57:35.286 INFO:teuthology.orchestra.run.smithi016.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T03:57:35.287 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T03:57:35.287 DEBUG:teuthology.orchestra.run.smithi016:> sudo rpm -qi grub2-tools 2024-03-26T03:57:35.328 INFO:teuthology.orchestra.run.smithi016.stdout:Name : grub2-tools 2024-03-26T03:57:35.328 INFO:teuthology.orchestra.run.smithi016.stdout:Epoch : 1 2024-03-26T03:57:35.328 INFO:teuthology.orchestra.run.smithi016.stdout:Version : 2.06 2024-03-26T03:57:35.328 INFO:teuthology.orchestra.run.smithi016.stdout:Release : 68.el9 2024-03-26T03:57:35.328 INFO:teuthology.orchestra.run.smithi016.stdout:Architecture: x86_64 2024-03-26T03:57:35.329 INFO:teuthology.orchestra.run.smithi016.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-26T03:57:35.329 INFO:teuthology.orchestra.run.smithi016.stdout:Group : Unspecified 2024-03-26T03:57:35.329 INFO:teuthology.orchestra.run.smithi016.stdout:Size : 8264254 2024-03-26T03:57:35.329 INFO:teuthology.orchestra.run.smithi016.stdout:License : GPLv3+ 2024-03-26T03:57:35.329 INFO:teuthology.orchestra.run.smithi016.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-26T03:57:35.329 INFO:teuthology.orchestra.run.smithi016.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-26T03:57:35.329 INFO:teuthology.orchestra.run.smithi016.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-26T03:57:35.329 INFO:teuthology.orchestra.run.smithi016.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-26T03:57:35.329 INFO:teuthology.orchestra.run.smithi016.stdout:Packager : builder@centos.org 2024-03-26T03:57:35.329 INFO:teuthology.orchestra.run.smithi016.stdout:Vendor : CentOS 2024-03-26T03:57:35.329 INFO:teuthology.orchestra.run.smithi016.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-26T03:57:35.329 INFO:teuthology.orchestra.run.smithi016.stdout:Summary : Support tools for GRUB. 2024-03-26T03:57:35.329 INFO:teuthology.orchestra.run.smithi016.stdout:Description : 2024-03-26T03:57:35.329 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T03:57:35.330 INFO:teuthology.orchestra.run.smithi016.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-26T03:57:35.330 INFO:teuthology.orchestra.run.smithi016.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-26T03:57:35.330 INFO:teuthology.orchestra.run.smithi016.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-26T03:57:35.330 INFO:teuthology.orchestra.run.smithi016.stdout:hardware devices. 2024-03-26T03:57:35.330 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T03:57:35.330 INFO:teuthology.orchestra.run.smithi016.stdout:This subpackage provides tools for support of all platforms. 2024-03-26T03:57:35.331 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-26T03:57:35.331 INFO:teuthology.task.kernel:Updating grub on smithi016 to boot 5.14.0-432.el9.x86_64 2024-03-26T03:57:35.331 DEBUG:teuthology.orchestra.run.smithi016:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-26T03:57:35.971 INFO:teuthology.orchestra.run.smithi016.stderr:Generating grub configuration file ... 2024-03-26T03:57:37.487 INFO:teuthology.orchestra.run.smithi016.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-26T03:57:37.537 INFO:teuthology.orchestra.run.smithi016.stderr:done 2024-03-26T03:57:37.545 DEBUG:teuthology.orchestra.run.smithi016:> mktemp 2024-03-26T03:57:37.560 INFO:teuthology.orchestra.run.smithi016.stdout:/tmp/tmp.6uqwYzS3Pt 2024-03-26T03:57:37.560 DEBUG:teuthology.orchestra.run.smithi016:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.6uqwYzS3Pt 2024-03-26T03:57:37.631 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0666 /tmp/tmp.6uqwYzS3Pt 2024-03-26T03:57:37.773 DEBUG:teuthology.orchestra.remote:smithi016:/tmp/tmp.6uqwYzS3Pt is 6KB 2024-03-26T03:57:37.823 DEBUG:teuthology.orchestra.run.smithi016:> rm -fr /tmp/tmp.6uqwYzS3Pt 2024-03-26T03:57:37.838 DEBUG:teuthology.orchestra.run.smithi016:> sudo /bin/ls /boot/loader/entries || true 2024-03-26T03:57:37.906 INFO:teuthology.orchestra.run.smithi016.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-26T03:57:37.906 INFO:teuthology.orchestra.run.smithi016.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-26T03:57:37.906 INFO:teuthology.orchestra.run.smithi016.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-26T03:57:37.906 INFO:teuthology.orchestra.run.smithi016.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-26T03:57:37.907 DEBUG:teuthology.orchestra.run.smithi016:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-26T03:57:38.047 DEBUG:teuthology.orchestra.run.smithi016:> sudo shutdown -r now 2024-03-26T03:57:43.838 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T03:57:43.839 INFO:teuthology.orchestra.run.smithi183.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-26T03:57:43.839 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T03:57:43.883 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-26T03:57:44.290 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-26T03:57:44.291 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-26T03:57:44.291 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-26T03:57:44.291 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-26T03:57:44.741 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-26T03:57:44.741 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T03:57:44.741 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-26T03:57:44.741 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-26T03:57:44.741 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-26T03:57:44.742 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-26T03:57:44.742 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-26T03:57:44.742 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T03:57:44.742 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-26T03:57:45.091 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T03:57:45.214 INFO:teuthology.orchestra.run.smithi183.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T03:57:45.215 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T03:57:45.215 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-26T03:57:45.215 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-26T03:57:45.215 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-26T03:57:45.215 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-26T03:57:45.215 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi183.front.sepia.ceph.com, path=None, version=distro) 2024-03-26T03:57:45.215 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q kernel | sort -rV | head -n 1 2024-03-26T03:57:45.294 INFO:teuthology.orchestra.run.smithi183.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-26T03:57:45.295 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-26T03:57:45.295 DEBUG:teuthology.orchestra.run.smithi183:> sudo rpm -qi grub2-tools 2024-03-26T03:57:45.329 INFO:teuthology.orchestra.run.smithi183.stdout:Name : grub2-tools 2024-03-26T03:57:45.329 INFO:teuthology.orchestra.run.smithi183.stdout:Epoch : 1 2024-03-26T03:57:45.329 INFO:teuthology.orchestra.run.smithi183.stdout:Version : 2.06 2024-03-26T03:57:45.329 INFO:teuthology.orchestra.run.smithi183.stdout:Release : 68.el9 2024-03-26T03:57:45.329 INFO:teuthology.orchestra.run.smithi183.stdout:Architecture: x86_64 2024-03-26T03:57:45.330 INFO:teuthology.orchestra.run.smithi183.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-26T03:57:45.330 INFO:teuthology.orchestra.run.smithi183.stdout:Group : Unspecified 2024-03-26T03:57:45.330 INFO:teuthology.orchestra.run.smithi183.stdout:Size : 8264254 2024-03-26T03:57:45.330 INFO:teuthology.orchestra.run.smithi183.stdout:License : GPLv3+ 2024-03-26T03:57:45.330 INFO:teuthology.orchestra.run.smithi183.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-26T03:57:45.330 INFO:teuthology.orchestra.run.smithi183.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-26T03:57:45.330 INFO:teuthology.orchestra.run.smithi183.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-26T03:57:45.330 INFO:teuthology.orchestra.run.smithi183.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-26T03:57:45.330 INFO:teuthology.orchestra.run.smithi183.stdout:Packager : builder@centos.org 2024-03-26T03:57:45.330 INFO:teuthology.orchestra.run.smithi183.stdout:Vendor : CentOS 2024-03-26T03:57:45.330 INFO:teuthology.orchestra.run.smithi183.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-26T03:57:45.331 INFO:teuthology.orchestra.run.smithi183.stdout:Summary : Support tools for GRUB. 2024-03-26T03:57:45.331 INFO:teuthology.orchestra.run.smithi183.stdout:Description : 2024-03-26T03:57:45.331 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T03:57:45.331 INFO:teuthology.orchestra.run.smithi183.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-26T03:57:45.331 INFO:teuthology.orchestra.run.smithi183.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-26T03:57:45.331 INFO:teuthology.orchestra.run.smithi183.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-26T03:57:45.331 INFO:teuthology.orchestra.run.smithi183.stdout:hardware devices. 2024-03-26T03:57:45.331 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T03:57:45.331 INFO:teuthology.orchestra.run.smithi183.stdout:This subpackage provides tools for support of all platforms. 2024-03-26T03:57:45.332 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-26T03:57:45.332 INFO:teuthology.task.kernel:Updating grub on smithi183 to boot 5.14.0-432.el9.x86_64 2024-03-26T03:57:45.332 DEBUG:teuthology.orchestra.run.smithi183:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-26T03:57:45.896 INFO:teuthology.orchestra.run.smithi183.stderr:Generating grub configuration file ... 2024-03-26T03:57:47.352 INFO:teuthology.orchestra.run.smithi183.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-26T03:57:47.373 INFO:teuthology.orchestra.run.smithi183.stderr:done 2024-03-26T03:57:47.375 DEBUG:teuthology.orchestra.run.smithi183:> mktemp 2024-03-26T03:57:47.390 INFO:teuthology.orchestra.run.smithi183.stdout:/tmp/tmp.R3bShdsHlu 2024-03-26T03:57:47.390 DEBUG:teuthology.orchestra.run.smithi183:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.R3bShdsHlu 2024-03-26T03:57:47.458 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0666 /tmp/tmp.R3bShdsHlu 2024-03-26T03:57:47.608 DEBUG:teuthology.orchestra.remote:smithi183:/tmp/tmp.R3bShdsHlu is 6KB 2024-03-26T03:57:47.655 DEBUG:teuthology.orchestra.run.smithi183:> rm -fr /tmp/tmp.R3bShdsHlu 2024-03-26T03:57:47.669 DEBUG:teuthology.orchestra.run.smithi183:> sudo /bin/ls /boot/loader/entries || true 2024-03-26T03:57:47.734 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-26T03:57:47.734 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-26T03:57:47.734 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-26T03:57:47.734 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-26T03:57:47.735 DEBUG:teuthology.orchestra.run.smithi183:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-26T03:57:47.897 DEBUG:teuthology.orchestra.run.smithi183:> sudo shutdown -r now 2024-03-26T03:58:08.070 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-26T03:58:08.071 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-03-26T03:58:08.072 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T03:58:17.910 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-26T03:58:17.911 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-26T03:58:17.911 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T03:58:26.591 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.16 2024-03-26T03:58:35.598 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-03-26T03:58:35.599 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T03:58:36.322 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-03-26T03:58:45.332 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-26T03:58:45.332 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T03:58:48.414 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-03-26T03:59:00.422 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-26T03:59:00.423 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T03:59:32.219 DEBUG:teuthology.orchestra.run.smithi183:> true 2024-03-26T03:59:32.736 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-26T03:59:32.736 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-432.el9.x86_64"... 2024-03-26T03:59:32.737 DEBUG:teuthology.orchestra.run.smithi183:> uname -r 2024-03-26T03:59:32.750 INFO:teuthology.orchestra.run.smithi183.stdout:5.14.0-432.el9.x86_64 2024-03-26T03:59:32.751 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-26T03:59:32.751 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-26T03:59:32.751 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-26T03:59:33.751 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-26T03:59:33.752 DEBUG:teuthology.orchestra.run.smithi183:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-26T03:59:33.822 INFO:teuthology.orchestra.run.smithi183.stdout:ttyS1 2024-03-26T03:59:33.854 DEBUG:teuthology.parallel:result is None 2024-03-26T03:59:35.602 DEBUG:teuthology.orchestra.remote:timed out 2024-03-26T03:59:47.603 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-03-26T03:59:47.604 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T03:59:47.913 DEBUG:teuthology.orchestra.run.smithi016:> true 2024-03-26T03:59:48.354 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-03-26T03:59:48.354 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-432.el9.x86_64"... 2024-03-26T03:59:48.354 DEBUG:teuthology.orchestra.run.smithi016:> uname -r 2024-03-26T03:59:48.370 INFO:teuthology.orchestra.run.smithi016.stdout:5.14.0-432.el9.x86_64 2024-03-26T03:59:48.371 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-26T03:59:48.371 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-26T03:59:48.371 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-26T03:59:49.372 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-26T03:59:49.372 DEBUG:teuthology.orchestra.run.smithi016:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-26T03:59:49.460 INFO:teuthology.orchestra.run.smithi016.stdout:ttyS1 2024-03-26T03:59:49.499 DEBUG:teuthology.parallel:result is None 2024-03-26T03:59:49.499 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-26T03:59:49.510 INFO:teuthology.task.internal:Creating test directory... 2024-03-26T03:59:49.511 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-26T03:59:49.513 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-26T03:59:49.532 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-26T03:59:49.539 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-26T03:59:49.547 INFO:teuthology.task.internal:Creating archive directory... 2024-03-26T03:59:49.547 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-26T03:59:49.569 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-26T03:59:49.648 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-26T03:59:49.657 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-26T03:59:49.657 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-26T03:59:49.660 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-26T03:59:49.694 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T03:59:49.695 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T03:59:49.708 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T03:59:49.709 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-26T03:59:49.711 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-26T03:59:49.718 INFO:teuthology.task.internal:Configuring sudo... 2024-03-26T03:59:49.718 DEBUG:teuthology.orchestra.run.smithi016:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-26T03:59:49.751 DEBUG:teuthology.orchestra.run.smithi183:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-26T03:59:49.781 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-26T03:59:49.791 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-26T03:59:49.791 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-26T03:59:49.818 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-26T03:59:49.837 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T03:59:49.912 DEBUG:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T03:59:49.963 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T03:59:50.043 DEBUG:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T03:59:50.087 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-26T03:59:50.087 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-26T03:59:50.157 DEBUG:teuthology.orchestra.run.smithi183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T03:59:50.201 DEBUG:teuthology.orchestra.run.smithi183:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-26T03:59:50.252 DEBUG:teuthology.orchestra.run.smithi183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T03:59:50.336 DEBUG:teuthology.orchestra.run.smithi183:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-26T03:59:50.382 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-26T03:59:50.382 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-26T03:59:50.452 DEBUG:teuthology.orchestra.run.smithi016:> sudo service rsyslog restart 2024-03-26T03:59:50.454 DEBUG:teuthology.orchestra.run.smithi183:> sudo service rsyslog restart 2024-03-26T03:59:50.511 INFO:teuthology.orchestra.run.smithi016.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T03:59:50.535 INFO:teuthology.orchestra.run.smithi183.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T03:59:50.828 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-26T03:59:50.835 INFO:teuthology.task.internal:Starting timer... 2024-03-26T03:59:50.835 INFO:teuthology.run_tasks:Running task pcp... 2024-03-26T03:59:50.848 INFO:teuthology.run_tasks:Running task selinux... 2024-03-26T03:59:50.857 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-26T03:59:50.857 DEBUG:teuthology.orchestra.run.smithi016:> sudo service auditd rotate 2024-03-26T03:59:50.912 INFO:teuthology.orchestra.run.smithi016.stdout:Rotating logs: 2024-03-26T03:59:50.914 DEBUG:teuthology.orchestra.run.smithi183:> sudo service auditd rotate 2024-03-26T03:59:50.970 INFO:teuthology.orchestra.run.smithi183.stdout:Rotating logs: 2024-03-26T03:59:50.972 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-26T03:59:50.972 DEBUG:teuthology.orchestra.run.smithi016:> /usr/sbin/getenforce 2024-03-26T03:59:51.008 INFO:teuthology.orchestra.run.smithi016.stdout:Permissive 2024-03-26T03:59:51.008 DEBUG:teuthology.orchestra.run.smithi183:> /usr/sbin/getenforce 2024-03-26T03:59:51.036 INFO:teuthology.orchestra.run.smithi183.stdout:Permissive 2024-03-26T03:59:51.037 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi016.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi183.front.sepia.ceph.com': 'permissive'} 2024-03-26T03:59:51.037 DEBUG:teuthology.orchestra.run.smithi016:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T03:59:51.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:51.076 DEBUG:teuthology.orchestra.run.smithi183:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T03:59:51.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T03:59:51.105 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-26T03:59:51.105 DEBUG:teuthology.orchestra.run.smithi016:> sudo /usr/sbin/setenforce permissive 2024-03-26T03:59:51.143 DEBUG:teuthology.orchestra.run.smithi183:> sudo /usr/sbin/setenforce permissive 2024-03-26T03:59:51.180 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-26T03:59:51.191 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-26T03:59:51.212 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-26T03:59:51.214 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi016.front.sepia.ceph.com,smithi183.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-26T04:04:08.980 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi016.front.sepia.ceph.com'), Remote(name='ubuntu@smithi183.front.sepia.ceph.com')] 2024-03-26T04:04:08.981 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-03-26T04:04:08.983 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:04:09.074 DEBUG:teuthology.orchestra.run.smithi016:> true 2024-03-26T04:04:09.148 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-03-26T04:04:09.149 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-26T04:04:09.149 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-26T04:04:09.249 DEBUG:teuthology.orchestra.run.smithi183:> true 2024-03-26T04:04:09.328 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-26T04:04:09.329 INFO:teuthology.run_tasks:Running task clock... 2024-03-26T04:04:09.343 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-26T04:04:09.343 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-26T04:04:09.344 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T04:04:09.347 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-26T04:04:09.347 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T04:04:09.376 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-26T04:04:09.393 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-26T04:04:09.399 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-26T04:04:09.416 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-26T04:04:09.422 INFO:teuthology.orchestra.run.smithi016.stderr:sudo: ntpd: command not found 2024-03-26T04:04:09.436 INFO:teuthology.orchestra.run.smithi016.stdout:506 Cannot talk to daemon 2024-03-26T04:04:09.445 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: ntpd: command not found 2024-03-26T04:04:09.452 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-26T04:04:09.458 INFO:teuthology.orchestra.run.smithi183.stdout:506 Cannot talk to daemon 2024-03-26T04:04:09.468 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-26T04:04:09.474 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-26T04:04:09.491 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-26T04:04:09.522 INFO:teuthology.orchestra.run.smithi016.stderr:bash: line 1: ntpq: command not found 2024-03-26T04:04:09.526 INFO:teuthology.orchestra.run.smithi016.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T04:04:09.526 INFO:teuthology.orchestra.run.smithi016.stdout:=============================================================================== 2024-03-26T04:04:09.526 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:04:09.526 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:04:09.526 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:04:09.527 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:04:09.542 INFO:teuthology.orchestra.run.smithi183.stderr:bash: line 1: ntpq: command not found 2024-03-26T04:04:09.546 INFO:teuthology.orchestra.run.smithi183.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T04:04:09.546 INFO:teuthology.orchestra.run.smithi183.stdout:=============================================================================== 2024-03-26T04:04:09.546 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:04:09.546 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:04:09.546 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:04:09.546 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:04:09.547 INFO:teuthology.run_tasks:Running task install... 2024-03-26T04:04:09.555 DEBUG:teuthology.task.install:project ceph 2024-03-26T04:04:09.555 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5e70124b46c2a671efcaae4690cd9ea72670710a'}, 'extra_packages': ['cephadm']} 2024-03-26T04:04:09.555 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5e70124b46c2a671efcaae4690cd9ea72670710a'} 2024-03-26T04:04:09.555 INFO:teuthology.task.install:Using flavor: default 2024-03-26T04:04:09.561 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-26T04:04:09.561 INFO:teuthology.task.install:extra packages: [] 2024-03-26T04:04:09.561 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5e70124b46c2a671efcaae4690cd9ea72670710a', 'tag': None, 'wait_for_package': False} 2024-03-26T04:04:09.562 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:04:09.564 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5e70124b46c2a671efcaae4690cd9ea72670710a', 'tag': None, 'wait_for_package': False} 2024-03-26T04:04:09.564 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:04:09.723 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/ 2024-03-26T04:04:09.723 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1775.g5e70124b 2024-03-26T04:04:09.799 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/ 2024-03-26T04:04:09.799 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1775.g5e70124b 2024-03-26T04:04:09.905 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-26T04:04:09.905 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-26T04:04:09.905 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-26T04:04:09.910 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/5e70124b46c2a671efcaae4690cd9ea72670710a/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-26T04:04:09.910 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-26T04:04:09.911 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-26T04:04:09.933 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-26T04:04:09.933 DEBUG:teuthology.orchestra.run.smithi183:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5e70124b46c2a671efcaae4690cd9ea72670710a/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-26T04:04:09.937 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-26T04:04:09.938 DEBUG:teuthology.orchestra.run.smithi016:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5e70124b46c2a671efcaae4690cd9ea72670710a/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-26T04:04:10.001 DEBUG:teuthology.orchestra.run.smithi183:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-26T04:04:10.005 DEBUG:teuthology.orchestra.run.smithi016:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-26T04:04:10.081 DEBUG:teuthology.orchestra.run.smithi183:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-26T04:04:10.085 DEBUG:teuthology.orchestra.run.smithi016:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-26T04:04:10.147 INFO:teuthology.orchestra.run.smithi183.stdout:check_obsoletes = 1 2024-03-26T04:04:10.149 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum clean all 2024-03-26T04:04:10.151 INFO:teuthology.orchestra.run.smithi016.stdout:check_obsoletes = 1 2024-03-26T04:04:10.152 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum clean all 2024-03-26T04:04:10.400 INFO:teuthology.orchestra.run.smithi183.stdout:74 files removed 2024-03-26T04:04:10.415 INFO:teuthology.orchestra.run.smithi016.stdout:74 files removed 2024-03-26T04:04:10.427 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-radosgw 2024-03-26T04:04:10.442 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-radosgw 2024-03-26T04:04:11.131 INFO:teuthology.orchestra.run.smithi183.stdout:ceph packages for x86_64 205 kB/s | 83 kB 00:00 2024-03-26T04:04:11.152 INFO:teuthology.orchestra.run.smithi016.stdout:ceph packages for x86_64 204 kB/s | 83 kB 00:00 2024-03-26T04:04:11.474 INFO:teuthology.orchestra.run.smithi183.stdout:ceph noarch packages 34 kB/s | 10 kB 00:00 2024-03-26T04:04:11.519 INFO:teuthology.orchestra.run.smithi016.stdout:ceph noarch packages 32 kB/s | 10 kB 00:00 2024-03-26T04:04:11.827 INFO:teuthology.orchestra.run.smithi183.stdout:ceph source packages 6.1 kB/s | 1.9 kB 00:00 2024-03-26T04:04:11.970 INFO:teuthology.orchestra.run.smithi016.stdout:ceph source packages 4.6 kB/s | 1.9 kB 00:00 2024-03-26T04:04:12.368 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-26T04:04:12.622 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-26T04:04:13.902 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - BaseOS 4.8 MB/s | 2.0 MB 00:00 2024-03-26T04:04:14.103 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS-9 - BaseOS 6.3 MB/s | 2.0 MB 00:00 2024-03-26T04:04:15.026 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-03-26T04:04:15.097 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-03-26T04:04:17.010 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-03-26T04:04:19.506 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - AppStream 5.8 MB/s | 19 MB 00:03 2024-03-26T04:04:20.870 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.1 MB 00:00 2024-03-26T04:04:22.428 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - Extras packages 73 kB/s | 16 kB 00:00 2024-03-26T04:04:23.697 INFO:teuthology.orchestra.run.smithi183.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-03-26T04:04:24.440 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - CRB 4.9 MB/s | 6.1 MB 00:01 2024-03-26T04:04:25.933 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS Stream 9 - Extras packages 57 kB/s | 16 kB 00:00 2024-03-26T04:04:27.529 INFO:teuthology.orchestra.run.smithi016.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-26T04:04:28.630 INFO:teuthology.orchestra.run.smithi183.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-26T04:04:31.116 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-26T04:04:31.134 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-26T04:04:31.134 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-26T04:04:31.134 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-26T04:04:31.135 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-26T04:04:31.135 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-radosgw x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 9.8 M 2024-03-26T04:04:31.135 INFO:teuthology.orchestra.run.smithi183.stdout:Upgrading: 2024-03-26T04:04:31.135 INFO:teuthology.orchestra.run.smithi183.stdout: librados2 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 3.4 M 2024-03-26T04:04:31.135 INFO:teuthology.orchestra.run.smithi183.stdout: librbd1 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 3.1 M 2024-03-26T04:04:31.135 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-26T04:04:31.135 INFO:teuthology.orchestra.run.smithi183.stdout: boost-program-options 2024-03-26T04:04:31.135 INFO:teuthology.orchestra.run.smithi183.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-26T04:04:31.135 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-base x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 5.3 M 2024-03-26T04:04:31.135 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-common x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 20 M 2024-03-26T04:04:31.135 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-selinux x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 25 k 2024-03-26T04:04:31.136 INFO:teuthology.orchestra.run.smithi183.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-26T04:04:31.136 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-26T04:04:31.136 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-26T04:04:31.136 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs2 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 744 k 2024-03-26T04:04:31.136 INFO:teuthology.orchestra.run.smithi183.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-26T04:04:31.136 INFO:teuthology.orchestra.run.smithi183.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-26T04:04:31.136 INFO:teuthology.orchestra.run.smithi183.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-26T04:04:31.136 INFO:teuthology.orchestra.run.smithi183.stdout: libradosstriper1 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 498 k 2024-03-26T04:04:31.136 INFO:teuthology.orchestra.run.smithi183.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-26T04:04:31.136 INFO:teuthology.orchestra.run.smithi183.stdout: librgw2 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 5.1 M 2024-03-26T04:04:31.136 INFO:teuthology.orchestra.run.smithi183.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-26T04:04:31.136 INFO:teuthology.orchestra.run.smithi183.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-26T04:04:31.137 INFO:teuthology.orchestra.run.smithi183.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-26T04:04:31.137 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-argparse 2024-03-26T04:04:31.137 INFO:teuthology.orchestra.run.smithi183.stdout: x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 45 k 2024-03-26T04:04:31.137 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-common x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 131 k 2024-03-26T04:04:31.137 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cephfs x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 162 k 2024-03-26T04:04:31.137 INFO:teuthology.orchestra.run.smithi183.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-26T04:04:31.137 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rados x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 321 k 2024-03-26T04:04:31.137 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rbd x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 298 k 2024-03-26T04:04:31.137 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rgw x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 99 k 2024-03-26T04:04:31.137 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-26T04:04:31.137 INFO:teuthology.orchestra.run.smithi183.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-26T04:04:31.138 INFO:teuthology.orchestra.run.smithi183.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-26T04:04:31.138 INFO:teuthology.orchestra.run.smithi183.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-26T04:04:31.138 INFO:teuthology.orchestra.run.smithi183.stdout:Installing weak dependencies: 2024-03-26T04:04:31.138 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-26T04:04:31.138 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:04:31.138 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-26T04:04:31.138 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-26T04:04:31.138 INFO:teuthology.orchestra.run.smithi183.stdout:Install 30 Packages 2024-03-26T04:04:31.138 INFO:teuthology.orchestra.run.smithi183.stdout:Upgrade 2 Packages 2024-03-26T04:04:31.138 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:04:31.139 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 58 M 2024-03-26T04:04:31.139 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-26T04:04:32.039 INFO:teuthology.orchestra.run.smithi183.stdout:(1/32): ceph-base-19.0.0-1775.g5e70124b.el9.x86 6.7 MB/s | 5.3 MB 00:00 2024-03-26T04:04:32.081 INFO:teuthology.orchestra.run.smithi183.stdout:(2/32): ceph-selinux-19.0.0-1775.g5e70124b.el9. 609 kB/s | 25 kB 00:00 2024-03-26T04:04:32.173 INFO:teuthology.orchestra.run.smithi183.stdout:(3/32): libcephfs2-19.0.0-1775.g5e70124b.el9.x8 7.9 MB/s | 744 kB 00:00 2024-03-26T04:04:32.273 INFO:teuthology.orchestra.run.smithi183.stdout:(4/32): libradosstriper1-19.0.0-1775.g5e70124b. 4.9 MB/s | 498 kB 00:00 2024-03-26T04:04:32.440 INFO:teuthology.orchestra.run.smithi183.stdout:(5/32): ceph-radosgw-19.0.0-1775.g5e70124b.el9. 8.2 MB/s | 9.8 MB 00:01 2024-03-26T04:04:32.499 INFO:teuthology.orchestra.run.smithi183.stdout:(6/32): python3-ceph-argparse-19.0.0-1775.g5e70 772 kB/s | 45 kB 00:00 2024-03-26T04:04:32.565 INFO:teuthology.orchestra.run.smithi183.stdout:(7/32): python3-ceph-common-19.0.0-1775.g5e7012 1.9 MB/s | 131 kB 00:00 2024-03-26T04:04:32.624 INFO:teuthology.orchestra.run.smithi183.stdout:(8/32): python3-cephfs-19.0.0-1775.g5e70124b.el 2.7 MB/s | 162 kB 00:00 2024-03-26T04:04:32.635 INFO:teuthology.orchestra.run.smithi016.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-03-26T04:04:32.691 INFO:teuthology.orchestra.run.smithi183.stdout:(9/32): python3-rados-19.0.0-1775.g5e70124b.el9 4.7 MB/s | 321 kB 00:00 2024-03-26T04:04:32.767 INFO:teuthology.orchestra.run.smithi183.stdout:(10/32): librgw2-19.0.0-1775.g5e70124b.el9.x86_ 10 MB/s | 5.1 MB 00:00 2024-03-26T04:04:32.792 INFO:teuthology.orchestra.run.smithi183.stdout:(11/32): python3-rbd-19.0.0-1775.g5e70124b.el9. 2.9 MB/s | 298 kB 00:00 2024-03-26T04:04:32.817 INFO:teuthology.orchestra.run.smithi183.stdout:(12/32): python3-rgw-19.0.0-1775.g5e70124b.el9. 1.9 MB/s | 99 kB 00:00 2024-03-26T04:04:32.959 INFO:teuthology.orchestra.run.smithi183.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 634 kB/s | 106 kB 00:00 2024-03-26T04:04:32.984 INFO:teuthology.orchestra.run.smithi183.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 159 kB 00:00 2024-03-26T04:04:33.001 INFO:teuthology.orchestra.run.smithi183.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-26T04:04:33.060 INFO:teuthology.orchestra.run.smithi183.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-26T04:04:33.085 INFO:teuthology.orchestra.run.smithi183.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.3 MB/s | 282 kB 00:00 2024-03-26T04:04:33.102 INFO:teuthology.orchestra.run.smithi183.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 993 kB/s | 41 kB 00:00 2024-03-26T04:04:33.177 INFO:teuthology.orchestra.run.smithi183.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 480 kB/s | 36 kB 00:00 2024-03-26T04:04:33.227 INFO:teuthology.orchestra.run.smithi183.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 950 kB/s | 47 kB 00:00 2024-03-26T04:04:33.253 INFO:teuthology.orchestra.run.smithi183.stdout:(21/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.3 MB/s | 221 kB 00:00 2024-03-26T04:04:33.703 INFO:teuthology.orchestra.run.smithi183.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 650 kB/s | 309 kB 00:00 2024-03-26T04:04:33.795 INFO:teuthology.orchestra.run.smithi183.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 283 kB/s | 26 kB 00:00 2024-03-26T04:04:33.887 INFO:teuthology.orchestra.run.smithi183.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 531 kB/s | 49 kB 00:00 2024-03-26T04:04:33.987 INFO:teuthology.orchestra.run.smithi183.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 674 kB/s | 67 kB 00:00 2024-03-26T04:04:34.071 INFO:teuthology.orchestra.run.smithi183.stdout:(26/32): libarrow-9.0.0-10.el9.x86_64.rpm 5.4 MB/s | 4.4 MB 00:00 2024-03-26T04:04:34.321 INFO:teuthology.orchestra.run.smithi183.stdout:(27/32): ceph-common-19.0.0-1775.g5e70124b.el9. 6.6 MB/s | 20 MB 00:03 2024-03-26T04:04:34.346 INFO:teuthology.orchestra.run.smithi183.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 696 kB/s | 192 kB 00:00 2024-03-26T04:04:34.380 INFO:teuthology.orchestra.run.smithi183.stdout:(29/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.1 MB/s | 838 kB 00:00 2024-03-26T04:04:34.872 INFO:teuthology.orchestra.run.smithi183.stdout:(30/32): librbd1-19.0.0-1775.g5e70124b.el9.x86_ 6.4 MB/s | 3.1 MB 00:00 2024-03-26T04:04:35.047 INFO:teuthology.orchestra.run.smithi183.stdout:(31/32): thrift-0.15.0-2.el9.x86_64.rpm 2.2 MB/s | 1.6 MB 00:00 2024-03-26T04:04:35.130 INFO:teuthology.orchestra.run.smithi183.stdout:(32/32): librados2-19.0.0-1775.g5e70124b.el9.x8 4.3 MB/s | 3.4 MB 00:00 2024-03-26T04:04:35.137 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:04:35.137 INFO:teuthology.orchestra.run.smithi183.stdout:Total 15 MB/s | 58 MB 00:03 2024-03-26T04:04:35.171 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-26T04:04:35.186 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-26T04:04:35.189 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-26T04:04:35.189 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-03-26T04:04:35.189 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-26T04:04:35.189 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-26T04:04:35.189 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-radosgw x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 9.8 M 2024-03-26T04:04:35.189 INFO:teuthology.orchestra.run.smithi016.stdout:Upgrading: 2024-03-26T04:04:35.189 INFO:teuthology.orchestra.run.smithi016.stdout: librados2 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 3.4 M 2024-03-26T04:04:35.189 INFO:teuthology.orchestra.run.smithi016.stdout: librbd1 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 3.1 M 2024-03-26T04:04:35.189 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-03-26T04:04:35.189 INFO:teuthology.orchestra.run.smithi016.stdout: boost-program-options 2024-03-26T04:04:35.190 INFO:teuthology.orchestra.run.smithi016.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-26T04:04:35.190 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 5.3 M 2024-03-26T04:04:35.190 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-common x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 20 M 2024-03-26T04:04:35.190 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-selinux x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 25 k 2024-03-26T04:04:35.190 INFO:teuthology.orchestra.run.smithi016.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-26T04:04:35.190 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-26T04:04:35.190 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-26T04:04:35.190 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs2 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 744 k 2024-03-26T04:04:35.190 INFO:teuthology.orchestra.run.smithi016.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-26T04:04:35.190 INFO:teuthology.orchestra.run.smithi016.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-26T04:04:35.190 INFO:teuthology.orchestra.run.smithi016.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-26T04:04:35.191 INFO:teuthology.orchestra.run.smithi016.stdout: libradosstriper1 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 498 k 2024-03-26T04:04:35.191 INFO:teuthology.orchestra.run.smithi016.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-26T04:04:35.191 INFO:teuthology.orchestra.run.smithi016.stdout: librgw2 x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 5.1 M 2024-03-26T04:04:35.191 INFO:teuthology.orchestra.run.smithi016.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-26T04:04:35.191 INFO:teuthology.orchestra.run.smithi016.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-26T04:04:35.191 INFO:teuthology.orchestra.run.smithi016.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-26T04:04:35.191 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-argparse 2024-03-26T04:04:35.191 INFO:teuthology.orchestra.run.smithi016.stdout: x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 45 k 2024-03-26T04:04:35.191 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-common x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 131 k 2024-03-26T04:04:35.191 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cephfs x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 162 k 2024-03-26T04:04:35.191 INFO:teuthology.orchestra.run.smithi016.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-26T04:04:35.191 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rados x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 321 k 2024-03-26T04:04:35.191 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rbd x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 298 k 2024-03-26T04:04:35.191 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rgw x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 99 k 2024-03-26T04:04:35.191 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-26T04:04:35.192 INFO:teuthology.orchestra.run.smithi016.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-26T04:04:35.192 INFO:teuthology.orchestra.run.smithi016.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-26T04:04:35.192 INFO:teuthology.orchestra.run.smithi016.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-26T04:04:35.192 INFO:teuthology.orchestra.run.smithi016.stdout:Installing weak dependencies: 2024-03-26T04:04:35.192 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-26T04:04:35.192 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:04:35.192 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-26T04:04:35.192 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-26T04:04:35.192 INFO:teuthology.orchestra.run.smithi016.stdout:Install 30 Packages 2024-03-26T04:04:35.192 INFO:teuthology.orchestra.run.smithi016.stdout:Upgrade 2 Packages 2024-03-26T04:04:35.192 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:04:35.193 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 58 M 2024-03-26T04:04:35.193 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-26T04:04:35.209 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-26T04:04:35.209 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-26T04:04:35.698 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-26T04:04:35.698 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-26T04:04:36.361 INFO:teuthology.orchestra.run.smithi016.stdout:(1/32): ceph-base-19.0.0-1775.g5e70124b.el9.x86 5.4 MB/s | 5.3 MB 00:00 2024-03-26T04:04:36.430 INFO:teuthology.orchestra.run.smithi016.stdout:(2/32): ceph-selinux-19.0.0-1775.g5e70124b.el9. 378 kB/s | 25 kB 00:00 2024-03-26T04:04:36.611 INFO:teuthology.orchestra.run.smithi016.stdout:(3/32): libcephfs2-19.0.0-1775.g5e70124b.el9.x8 4.0 MB/s | 744 kB 00:00 2024-03-26T04:04:36.954 INFO:teuthology.orchestra.run.smithi016.stdout:(4/32): libradosstriper1-19.0.0-1775.g5e70124b. 1.4 MB/s | 498 kB 00:00 2024-03-26T04:04:37.087 INFO:teuthology.orchestra.run.smithi016.stdout:(5/32): ceph-radosgw-19.0.0-1775.g5e70124b.el9. 5.7 MB/s | 9.8 MB 00:01 2024-03-26T04:04:37.137 INFO:teuthology.orchestra.run.smithi016.stdout:(6/32): python3-ceph-argparse-19.0.0-1775.g5e70 904 kB/s | 45 kB 00:00 2024-03-26T04:04:37.144 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-26T04:04:37.196 INFO:teuthology.orchestra.run.smithi016.stdout:(7/32): python3-ceph-common-19.0.0-1775.g5e7012 2.2 MB/s | 131 kB 00:00 2024-03-26T04:04:37.220 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-26T04:04:37.252 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-26T04:04:37.254 INFO:teuthology.orchestra.run.smithi016.stdout:(8/32): python3-cephfs-19.0.0-1775.g5e70124b.el 2.7 MB/s | 162 kB 00:00 2024-03-26T04:04:37.277 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-26T04:04:37.300 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-26T04:04:37.317 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-26T04:04:37.338 INFO:teuthology.orchestra.run.smithi016.stdout:(9/32): python3-rados-19.0.0-1775.g5e70124b.el9 3.8 MB/s | 321 kB 00:00 2024-03-26T04:04:37.405 INFO:teuthology.orchestra.run.smithi016.stdout:(10/32): python3-rbd-19.0.0-1775.g5e70124b.el9. 4.4 MB/s | 298 kB 00:00 2024-03-26T04:04:37.456 INFO:teuthology.orchestra.run.smithi016.stdout:(11/32): python3-rgw-19.0.0-1775.g5e70124b.el9. 1.9 MB/s | 99 kB 00:00 2024-03-26T04:04:37.458 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-26T04:04:37.513 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-26T04:04:37.590 INFO:teuthology.orchestra.run.smithi016.stdout:(12/32): librgw2-19.0.0-1775.g5e70124b.el9.x86_ 8.1 MB/s | 5.1 MB 00:00 2024-03-26T04:04:37.623 INFO:teuthology.orchestra.run.smithi016.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 632 kB/s | 106 kB 00:00 2024-03-26T04:04:37.673 INFO:teuthology.orchestra.run.smithi016.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 159 kB 00:00 2024-03-26T04:04:37.698 INFO:teuthology.orchestra.run.smithi016.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 589 kB/s | 44 kB 00:00 2024-03-26T04:04:37.882 INFO:teuthology.orchestra.run.smithi016.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 3.1 MB/s | 661 kB 00:00 2024-03-26T04:04:37.896 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-26T04:04:37.928 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 7/34 2024-03-26T04:04:37.974 INFO:teuthology.orchestra.run.smithi016.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.0 MB/s | 282 kB 00:00 2024-03-26T04:04:37.999 INFO:teuthology.orchestra.run.smithi016.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 352 kB/s | 41 kB 00:00 2024-03-26T04:04:38.004 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 7/34 2024-03-26T04:04:38.094 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rados-2:19.0.0-1775.g5e70124b.el9.x86_64 8/34 2024-03-26T04:04:38.121 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 9/34 2024-03-26T04:04:38.139 INFO:teuthology.orchestra.run.smithi016.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-26T04:04:38.140 INFO:teuthology.orchestra.run.smithi016.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-26T04:04:38.184 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 9/34 2024-03-26T04:04:38.218 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-26T04:04:38.283 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ceph-argparse-2:19.0.0-1775.g5e70124b.el9. 11/34 2024-03-26T04:04:38.313 INFO:teuthology.orchestra.run.smithi016.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-26T04:04:38.339 INFO:teuthology.orchestra.run.smithi016.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-26T04:04:38.383 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cephfs-2:19.0.0-1775.g5e70124b.el9.x86_64 12/34 2024-03-26T04:04:38.416 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libradosstriper1-2:19.0.0-1775.g5e70124b.el9.x86_6 13/34 2024-03-26T04:04:38.483 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1775.g5e70124b.el9.x86_6 13/34 2024-03-26T04:04:38.517 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-26T04:04:38.575 INFO:teuthology.orchestra.run.smithi016.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 62 kB/s | 36 kB 00:00 2024-03-26T04:04:38.607 INFO:teuthology.orchestra.run.smithi016.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-26T04:04:38.641 INFO:teuthology.orchestra.run.smithi016.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-26T04:04:38.684 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-26T04:04:38.718 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-26T04:04:38.764 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-26T04:04:38.798 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-26T04:04:38.829 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-26T04:04:38.833 INFO:teuthology.orchestra.run.smithi016.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 183 kB/s | 47 kB 00:00 2024-03-26T04:04:38.861 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-26T04:04:38.883 INFO:teuthology.orchestra.run.smithi016.stdout:(21/32): qatlib-24.02.0-1.el9.x86_64.rpm 243 kB/s | 221 kB 00:00 2024-03-26T04:04:39.001 INFO:teuthology.orchestra.run.smithi016.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.8 MB/s | 309 kB 00:00 2024-03-26T04:04:39.043 INFO:teuthology.orchestra.run.smithi016.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 623 kB/s | 26 kB 00:00 2024-03-26T04:04:39.094 INFO:teuthology.orchestra.run.smithi016.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 977 kB/s | 49 kB 00:00 2024-03-26T04:04:39.135 INFO:teuthology.orchestra.run.smithi016.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-03-26T04:04:39.255 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-26T04:04:39.288 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 22/34 2024-03-26T04:04:39.371 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 22/34 2024-03-26T04:04:39.401 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rbd-2:19.0.0-1775.g5e70124b.el9.x86_64 23/34 2024-03-26T04:04:39.444 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-26T04:04:39.460 INFO:teuthology.orchestra.run.smithi016.stdout:(26/32): ceph-common-19.0.0-1775.g5e70124b.el9. 5.0 MB/s | 20 MB 00:04 2024-03-26T04:04:39.502 INFO:teuthology.orchestra.run.smithi016.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.2 MB/s | 838 kB 00:00 2024-03-26T04:04:39.610 INFO:teuthology.orchestra.run.smithi016.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-26T04:04:39.836 INFO:teuthology.orchestra.run.smithi016.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 4.6 MB/s | 4.4 MB 00:00 2024-03-26T04:04:39.878 INFO:teuthology.orchestra.run.smithi016.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 4.2 MB/s | 1.6 MB 00:00 2024-03-26T04:04:39.905 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-26T04:04:39.932 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librgw2-2:19.0.0-1775.g5e70124b.el9.x86_64 26/34 2024-03-26T04:04:40.011 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librgw2-2:19.0.0-1775.g5e70124b.el9.x86_64 26/34 2024-03-26T04:04:40.036 INFO:teuthology.orchestra.run.smithi016.stdout:(31/32): librados2-19.0.0-1775.g5e70124b.el9.x8 8.0 MB/s | 3.4 MB 00:00 2024-03-26T04:04:40.080 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rgw-2:19.0.0-1775.g5e70124b.el9.x86_64 27/34 2024-03-26T04:04:40.109 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ceph-common-2:19.0.0-1775.g5e70124b.el9.x8 28/34 2024-03-26T04:04:40.320 INFO:teuthology.orchestra.run.smithi016.stdout:(32/32): librbd1-19.0.0-1775.g5e70124b.el9.x86_ 6.5 MB/s | 3.1 MB 00:00 2024-03-26T04:04:40.325 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:04:40.326 INFO:teuthology.orchestra.run.smithi016.stdout:Total 11 MB/s | 58 MB 00:05 2024-03-26T04:04:40.360 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-26T04:04:40.402 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-26T04:04:40.402 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-26T04:04:40.905 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-26T04:04:40.906 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-26T04:04:42.338 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 29/34 2024-03-26T04:04:42.380 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 29/34 2024-03-26T04:04:42.437 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-26T04:04:42.523 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-26T04:04:42.558 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-26T04:04:42.583 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-26T04:04:42.606 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-26T04:04:42.623 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-26T04:04:42.815 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-26T04:04:42.871 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-26T04:04:42.937 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 29/34 2024-03-26T04:04:42.976 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 30/34 2024-03-26T04:04:43.067 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 30/34 2024-03-26T04:04:43.067 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-26T04:04:43.067 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-26T04:04:43.067 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:04:43.118 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-selinux-2:19.0.0-1775.g5e70124b.el9.x86_64 31/34 2024-03-26T04:04:43.214 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-26T04:04:43.260 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 7/34 2024-03-26T04:04:43.336 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 7/34 2024-03-26T04:04:43.427 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rados-2:19.0.0-1775.g5e70124b.el9.x86_64 8/34 2024-03-26T04:04:43.453 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 9/34 2024-03-26T04:04:43.499 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 9/34 2024-03-26T04:04:43.528 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-26T04:04:43.589 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ceph-argparse-2:19.0.0-1775.g5e70124b.el9. 11/34 2024-03-26T04:04:43.801 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cephfs-2:19.0.0-1775.g5e70124b.el9.x86_64 12/34 2024-03-26T04:04:43.831 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libradosstriper1-2:19.0.0-1775.g5e70124b.el9.x86_6 13/34 2024-03-26T04:04:43.878 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1775.g5e70124b.el9.x86_6 13/34 2024-03-26T04:04:43.913 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-26T04:04:44.023 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-26T04:04:44.055 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-26T04:04:44.096 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-26T04:04:44.120 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-26T04:04:44.149 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-26T04:04:44.182 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-26T04:04:44.506 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-26T04:04:44.537 INFO:teuthology.orchestra.run.smithi016.stdout: Upgrading : librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 22/34 2024-03-26T04:04:44.628 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 22/34 2024-03-26T04:04:44.658 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rbd-2:19.0.0-1775.g5e70124b.el9.x86_64 23/34 2024-03-26T04:04:44.706 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-26T04:04:45.196 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-26T04:04:45.223 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librgw2-2:19.0.0-1775.g5e70124b.el9.x86_64 26/34 2024-03-26T04:04:45.294 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librgw2-2:19.0.0-1775.g5e70124b.el9.x86_64 26/34 2024-03-26T04:04:45.365 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rgw-2:19.0.0-1775.g5e70124b.el9.x86_64 27/34 2024-03-26T04:04:45.401 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ceph-common-2:19.0.0-1775.g5e70124b.el9.x8 28/34 2024-03-26T04:04:47.649 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 29/34 2024-03-26T04:04:47.689 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 29/34 2024-03-26T04:04:48.271 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 29/34 2024-03-26T04:04:48.309 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 30/34 2024-03-26T04:04:48.390 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 30/34 2024-03-26T04:04:48.390 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-26T04:04:48.390 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-26T04:04:48.390 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:04:48.427 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-selinux-2:19.0.0-1775.g5e70124b.el9.x86_64 31/34 2024-03-26T04:04:52.459 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1775.g5e70124b.el9.x86_64 31/34 2024-03-26T04:04:52.460 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /sys 2024-03-26T04:04:52.460 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /proc 2024-03-26T04:04:52.460 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /mnt 2024-03-26T04:04:52.460 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /var/tmp 2024-03-26T04:04:52.460 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /home 2024-03-26T04:04:52.460 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /root 2024-03-26T04:04:52.460 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /tmp 2024-03-26T04:04:52.460 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:04:53.392 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-radosgw-2:19.0.0-1775.g5e70124b.el9.x86_64 32/34 2024-03-26T04:04:53.417 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1775.g5e70124b.el9.x86_64 32/34 2024-03-26T04:04:53.417 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:04:53.417 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-26T04:04:53.417 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T04:04:53.417 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T04:04:53.417 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:04:53.419 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-26T04:04:53.511 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-26T04:04:53.511 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T04:04:54.647 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T04:04:54.647 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 1/34 2024-03-26T04:04:54.647 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 2/34 2024-03-26T04:04:54.647 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-radosgw-2:19.0.0-1775.g5e70124b.el9.x86_64 3/34 2024-03-26T04:04:54.647 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-selinux-2:19.0.0-1775.g5e70124b.el9.x86_64 4/34 2024-03-26T04:04:54.647 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 5/34 2024-03-26T04:04:54.647 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libradosstriper1-2:19.0.0-1775.g5e70124b.el9.x86_6 6/34 2024-03-26T04:04:54.648 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librgw2-2:19.0.0-1775.g5e70124b.el9.x86_64 7/34 2024-03-26T04:04:54.648 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1775.g5e70124b.el9. 8/34 2024-03-26T04:04:54.648 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ceph-common-2:19.0.0-1775.g5e70124b.el9.x8 9/34 2024-03-26T04:04:54.648 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cephfs-2:19.0.0-1775.g5e70124b.el9.x86_64 10/34 2024-03-26T04:04:54.648 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rados-2:19.0.0-1775.g5e70124b.el9.x86_64 11/34 2024-03-26T04:04:54.648 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rbd-2:19.0.0-1775.g5e70124b.el9.x86_64 12/34 2024-03-26T04:04:54.648 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rgw-2:19.0.0-1775.g5e70124b.el9.x86_64 13/34 2024-03-26T04:04:54.648 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-26T04:04:54.648 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-26T04:04:54.648 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-26T04:04:54.648 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-26T04:04:54.648 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-26T04:04:54.648 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-26T04:04:54.648 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-26T04:04:54.649 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-26T04:04:54.649 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-26T04:04:54.649 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-26T04:04:54.649 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-26T04:04:54.649 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-26T04:04:54.649 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-26T04:04:54.649 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-26T04:04:54.649 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-26T04:04:54.649 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-26T04:04:54.649 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-26T04:04:54.649 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 31/34 2024-03-26T04:04:54.649 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-26T04:04:54.649 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 33/34 2024-03-26T04:04:55.753 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T04:04:55.753 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:04:55.753 INFO:teuthology.orchestra.run.smithi183.stdout:Upgraded: 2024-03-26T04:04:55.753 INFO:teuthology.orchestra.run.smithi183.stdout: librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:04:55.753 INFO:teuthology.orchestra.run.smithi183.stdout: librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:04:55.753 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-26T04:04:55.753 INFO:teuthology.orchestra.run.smithi183.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-26T04:04:55.753 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:04:55.753 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:04:55.753 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-radosgw-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:04:55.753 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-selinux-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:04:55.754 INFO:teuthology.orchestra.run.smithi183.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-26T04:04:55.754 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-26T04:04:55.754 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-26T04:04:55.754 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:04:55.754 INFO:teuthology.orchestra.run.smithi183.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-26T04:04:55.754 INFO:teuthology.orchestra.run.smithi183.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-26T04:04:55.754 INFO:teuthology.orchestra.run.smithi183.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-26T04:04:55.754 INFO:teuthology.orchestra.run.smithi183.stdout: libradosstriper1-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:04:55.754 INFO:teuthology.orchestra.run.smithi183.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-26T04:04:55.754 INFO:teuthology.orchestra.run.smithi183.stdout: librgw2-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:04:55.754 INFO:teuthology.orchestra.run.smithi183.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-26T04:04:55.754 INFO:teuthology.orchestra.run.smithi183.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-26T04:04:55.755 INFO:teuthology.orchestra.run.smithi183.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-26T04:04:55.755 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-argparse-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:04:55.755 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:04:55.755 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cephfs-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:04:55.755 INFO:teuthology.orchestra.run.smithi183.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-26T04:04:55.755 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rados-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:04:55.755 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rbd-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:04:55.755 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rgw-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:04:55.755 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-26T04:04:55.755 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-26T04:04:55.755 INFO:teuthology.orchestra.run.smithi183.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-26T04:04:55.755 INFO:teuthology.orchestra.run.smithi183.stdout: re2-1:20211101-3.el9.x86_64 2024-03-26T04:04:55.755 INFO:teuthology.orchestra.run.smithi183.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-26T04:04:55.755 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:04:55.755 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-26T04:04:56.042 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-test 2024-03-26T04:04:56.596 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:28 ago on Tue 26 Mar 2024 04:04:28 AM UTC. 2024-03-26T04:04:56.747 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-26T04:04:56.747 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-26T04:04:56.747 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-26T04:04:56.748 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-26T04:04:56.748 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-26T04:04:56.748 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-test x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 47 M 2024-03-26T04:04:56.748 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-26T04:04:56.748 INFO:teuthology.orchestra.run.smithi183.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-26T04:04:56.748 INFO:teuthology.orchestra.run.smithi183.stdout: libcephsqlite x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 163 k 2024-03-26T04:04:56.748 INFO:teuthology.orchestra.run.smithi183.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-26T04:04:56.748 INFO:teuthology.orchestra.run.smithi183.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-26T04:04:56.748 INFO:teuthology.orchestra.run.smithi183.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-26T04:04:56.748 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:04:56.748 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-26T04:04:56.748 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-26T04:04:56.748 INFO:teuthology.orchestra.run.smithi183.stdout:Install 6 Packages 2024-03-26T04:04:56.749 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:04:56.749 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 48 M 2024-03-26T04:04:56.749 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 205 M 2024-03-26T04:04:56.749 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-26T04:04:56.965 INFO:teuthology.orchestra.run.smithi183.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-26T04:04:57.048 INFO:teuthology.orchestra.run.smithi183.stdout:(2/6): libcephsqlite-19.0.0-1775.g5e70124b.el9. 673 kB/s | 163 kB 00:00 2024-03-26T04:04:57.074 INFO:teuthology.orchestra.run.smithi183.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 2.7 MB/s | 300 kB 00:00 2024-03-26T04:04:57.099 INFO:teuthology.orchestra.run.smithi183.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-26T04:04:57.166 INFO:teuthology.orchestra.run.smithi183.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 2.0 MB/s | 187 kB 00:00 2024-03-26T04:04:58.516 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1775.g5e70124b.el9.x86_64 31/34 2024-03-26T04:04:58.517 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /sys 2024-03-26T04:04:58.517 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /proc 2024-03-26T04:04:58.517 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /mnt 2024-03-26T04:04:58.517 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /var/tmp 2024-03-26T04:04:58.517 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /home 2024-03-26T04:04:58.517 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /root 2024-03-26T04:04:58.517 INFO:teuthology.orchestra.run.smithi016.stdout:skipping the directory /tmp 2024-03-26T04:04:58.517 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:04:59.459 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-radosgw-2:19.0.0-1775.g5e70124b.el9.x86_64 32/34 2024-03-26T04:04:59.483 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1775.g5e70124b.el9.x86_64 32/34 2024-03-26T04:04:59.483 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:04:59.483 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-26T04:04:59.483 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T04:04:59.483 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-26T04:04:59.483 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:04:59.485 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-26T04:04:59.578 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-26T04:04:59.578 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T04:04:59.782 INFO:teuthology.orchestra.run.smithi183.stdout:(6/6): ceph-test-19.0.0-1775.g5e70124b.el9.x86_ 16 MB/s | 47 MB 00:02 2024-03-26T04:04:59.785 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:04:59.786 INFO:teuthology.orchestra.run.smithi183.stdout:Total 16 MB/s | 48 MB 00:03 2024-03-26T04:04:59.801 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-26T04:04:59.822 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-26T04:04:59.822 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-26T04:05:00.206 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-26T04:05:00.206 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-26T04:05:00.660 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T04:05:00.661 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 1/34 2024-03-26T04:05:00.661 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 2/34 2024-03-26T04:05:00.661 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-radosgw-2:19.0.0-1775.g5e70124b.el9.x86_64 3/34 2024-03-26T04:05:00.661 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-selinux-2:19.0.0-1775.g5e70124b.el9.x86_64 4/34 2024-03-26T04:05:00.661 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 5/34 2024-03-26T04:05:00.661 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libradosstriper1-2:19.0.0-1775.g5e70124b.el9.x86_6 6/34 2024-03-26T04:05:00.661 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librgw2-2:19.0.0-1775.g5e70124b.el9.x86_64 7/34 2024-03-26T04:05:00.661 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1775.g5e70124b.el9. 8/34 2024-03-26T04:05:00.661 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ceph-common-2:19.0.0-1775.g5e70124b.el9.x8 9/34 2024-03-26T04:05:00.662 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cephfs-2:19.0.0-1775.g5e70124b.el9.x86_64 10/34 2024-03-26T04:05:00.662 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rados-2:19.0.0-1775.g5e70124b.el9.x86_64 11/34 2024-03-26T04:05:00.662 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rbd-2:19.0.0-1775.g5e70124b.el9.x86_64 12/34 2024-03-26T04:05:00.662 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rgw-2:19.0.0-1775.g5e70124b.el9.x86_64 13/34 2024-03-26T04:05:00.662 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-26T04:05:00.662 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-26T04:05:00.662 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-26T04:05:00.662 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-26T04:05:00.662 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-26T04:05:00.662 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-26T04:05:00.662 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-26T04:05:00.663 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-26T04:05:00.663 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-26T04:05:00.663 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-26T04:05:00.663 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-26T04:05:00.663 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-26T04:05:00.663 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-26T04:05:00.663 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-26T04:05:00.664 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-26T04:05:00.664 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-26T04:05:00.664 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-26T04:05:00.664 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 31/34 2024-03-26T04:05:00.664 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-26T04:05:00.664 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 33/34 2024-03-26T04:05:00.799 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-26T04:05:00.876 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-26T04:05:00.917 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-26T04:05:00.948 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-26T04:05:00.990 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-26T04:05:01.019 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephsqlite-2:19.0.0-1775.g5e70124b.el9.x86_64 5/6 2024-03-26T04:05:01.752 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-26T04:05:01.752 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:05:01.752 INFO:teuthology.orchestra.run.smithi016.stdout:Upgraded: 2024-03-26T04:05:01.752 INFO:teuthology.orchestra.run.smithi016.stdout: librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:05:01.752 INFO:teuthology.orchestra.run.smithi016.stdout: librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:05:01.752 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-26T04:05:01.752 INFO:teuthology.orchestra.run.smithi016.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-26T04:05:01.752 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:05:01.752 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:05:01.752 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-radosgw-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:05:01.752 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-selinux-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:05:01.752 INFO:teuthology.orchestra.run.smithi016.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-26T04:05:01.753 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-26T04:05:01.753 INFO:teuthology.orchestra.run.smithi016.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-26T04:05:01.753 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:05:01.753 INFO:teuthology.orchestra.run.smithi016.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-26T04:05:01.753 INFO:teuthology.orchestra.run.smithi016.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-26T04:05:01.753 INFO:teuthology.orchestra.run.smithi016.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-26T04:05:01.753 INFO:teuthology.orchestra.run.smithi016.stdout: libradosstriper1-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:05:01.753 INFO:teuthology.orchestra.run.smithi016.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-26T04:05:01.753 INFO:teuthology.orchestra.run.smithi016.stdout: librgw2-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:05:01.753 INFO:teuthology.orchestra.run.smithi016.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-26T04:05:01.753 INFO:teuthology.orchestra.run.smithi016.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-26T04:05:01.753 INFO:teuthology.orchestra.run.smithi016.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-26T04:05:01.753 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-argparse-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:05:01.753 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-common-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:05:01.754 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cephfs-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:05:01.754 INFO:teuthology.orchestra.run.smithi016.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-26T04:05:01.754 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rados-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:05:01.754 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rbd-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:05:01.754 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rgw-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:05:01.754 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-26T04:05:01.754 INFO:teuthology.orchestra.run.smithi016.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-26T04:05:01.754 INFO:teuthology.orchestra.run.smithi016.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-26T04:05:01.754 INFO:teuthology.orchestra.run.smithi016.stdout: re2-1:20211101-3.el9.x86_64 2024-03-26T04:05:01.754 INFO:teuthology.orchestra.run.smithi016.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-26T04:05:01.754 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:05:01.754 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-26T04:05:02.055 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-test 2024-03-26T04:05:02.617 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:00:30 ago on Tue 26 Mar 2024 04:04:32 AM UTC. 2024-03-26T04:05:02.774 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-26T04:05:02.774 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-26T04:05:02.774 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-03-26T04:05:02.774 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-26T04:05:02.775 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-26T04:05:02.775 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-test x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 47 M 2024-03-26T04:05:02.775 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-03-26T04:05:02.775 INFO:teuthology.orchestra.run.smithi016.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-26T04:05:02.775 INFO:teuthology.orchestra.run.smithi016.stdout: libcephsqlite x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 163 k 2024-03-26T04:05:02.775 INFO:teuthology.orchestra.run.smithi016.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-26T04:05:02.775 INFO:teuthology.orchestra.run.smithi016.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-26T04:05:02.775 INFO:teuthology.orchestra.run.smithi016.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-26T04:05:02.775 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:05:02.775 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-26T04:05:02.775 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-26T04:05:02.775 INFO:teuthology.orchestra.run.smithi016.stdout:Install 6 Packages 2024-03-26T04:05:02.775 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:05:02.776 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 48 M 2024-03-26T04:05:02.776 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 205 M 2024-03-26T04:05:02.776 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-26T04:05:03.040 INFO:teuthology.orchestra.run.smithi016.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.5 MB/s | 217 kB 00:00 2024-03-26T04:05:03.090 INFO:teuthology.orchestra.run.smithi016.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-26T04:05:03.123 INFO:teuthology.orchestra.run.smithi016.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 63 kB 00:00 2024-03-26T04:05:03.474 INFO:teuthology.orchestra.run.smithi016.stdout:(4/6): libcephsqlite-19.0.0-1775.g5e70124b.el9. 286 kB/s | 163 kB 00:00 2024-03-26T04:05:03.690 INFO:teuthology.orchestra.run.smithi016.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 330 kB/s | 187 kB 00:00 2024-03-26T04:05:05.424 INFO:teuthology.orchestra.run.smithi016.stdout:(6/6): ceph-test-19.0.0-1775.g5e70124b.el9.x86_ 19 MB/s | 47 MB 00:02 2024-03-26T04:05:05.427 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:05:05.428 INFO:teuthology.orchestra.run.smithi016.stdout:Total 18 MB/s | 48 MB 00:02 2024-03-26T04:05:05.442 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-26T04:05:05.466 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-26T04:05:05.466 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-26T04:05:05.533 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1775.g5e70124b.el9.x86_64 5/6 2024-03-26T04:05:05.610 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-test-2:19.0.0-1775.g5e70124b.el9.x86_64 6/6 2024-03-26T04:05:05.857 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-26T04:05:05.857 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-26T04:05:06.459 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-26T04:05:06.542 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-26T04:05:06.596 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-26T04:05:06.632 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-26T04:05:06.675 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-26T04:05:06.703 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephsqlite-2:19.0.0-1775.g5e70124b.el9.x86_64 5/6 2024-03-26T04:05:07.945 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-test-2:19.0.0-1775.g5e70124b.el9.x86_64 6/6 2024-03-26T04:05:07.946 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-test-2:19.0.0-1775.g5e70124b.el9.x86_64 1/6 2024-03-26T04:05:07.946 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephsqlite-2:19.0.0-1775.g5e70124b.el9.x86_64 2/6 2024-03-26T04:05:07.946 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-26T04:05:07.946 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-26T04:05:07.946 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-26T04:05:08.344 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-26T04:05:08.344 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:05:08.345 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-26T04:05:08.345 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-test-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:05:08.345 INFO:teuthology.orchestra.run.smithi183.stdout: jq-1.6-16.el9.x86_64 2024-03-26T04:05:08.345 INFO:teuthology.orchestra.run.smithi183.stdout: libcephsqlite-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:05:08.345 INFO:teuthology.orchestra.run.smithi183.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-26T04:05:08.345 INFO:teuthology.orchestra.run.smithi183.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-26T04:05:08.345 INFO:teuthology.orchestra.run.smithi183.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-26T04:05:08.345 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:05:08.345 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-26T04:05:08.645 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph 2024-03-26T04:05:09.219 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:41 ago on Tue 26 Mar 2024 04:04:28 AM UTC. 2024-03-26T04:05:09.383 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-26T04:05:09.385 INFO:teuthology.orchestra.run.smithi183.stdout:========================================================================================= 2024-03-26T04:05:09.385 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-26T04:05:09.386 INFO:teuthology.orchestra.run.smithi183.stdout:========================================================================================= 2024-03-26T04:05:09.386 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-26T04:05:09.386 INFO:teuthology.orchestra.run.smithi183.stdout: ceph x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 6.6 k 2024-03-26T04:05:09.386 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-26T04:05:09.386 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mds x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 2.3 M 2024-03-26T04:05:09.386 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 1.5 M 2024-03-26T04:05:09.386 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 248 k 2024-03-26T04:05:09.386 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mon x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 4.7 M 2024-03-26T04:05:09.386 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-osd x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 17 M 2024-03-26T04:05:09.386 INFO:teuthology.orchestra.run.smithi183.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-26T04:05:09.386 INFO:teuthology.orchestra.run.smithi183.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-26T04:05:09.386 INFO:teuthology.orchestra.run.smithi183.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-26T04:05:09.386 INFO:teuthology.orchestra.run.smithi183.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-26T04:05:09.387 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-26T04:05:09.387 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-26T04:05:09.387 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-26T04:05:09.387 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-26T04:05:09.387 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-26T04:05:09.387 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-26T04:05:09.387 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-26T04:05:09.387 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-26T04:05:09.387 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-26T04:05:09.387 INFO:teuthology.orchestra.run.smithi183.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-26T04:05:09.387 INFO:teuthology.orchestra.run.smithi183.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-26T04:05:09.387 INFO:teuthology.orchestra.run.smithi183.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-26T04:05:09.387 INFO:teuthology.orchestra.run.smithi183.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-26T04:05:09.388 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-26T04:05:09.388 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-26T04:05:09.388 INFO:teuthology.orchestra.run.smithi183.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-26T04:05:09.388 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-26T04:05:09.388 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-26T04:05:09.388 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-26T04:05:09.388 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-26T04:05:09.388 INFO:teuthology.orchestra.run.smithi183.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-26T04:05:09.388 INFO:teuthology.orchestra.run.smithi183.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-26T04:05:09.388 INFO:teuthology.orchestra.run.smithi183.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-26T04:05:09.388 INFO:teuthology.orchestra.run.smithi183.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-26T04:05:09.388 INFO:teuthology.orchestra.run.smithi183.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-26T04:05:09.388 INFO:teuthology.orchestra.run.smithi183.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-26T04:05:09.390 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:05:09.390 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-26T04:05:09.390 INFO:teuthology.orchestra.run.smithi183.stdout:========================================================================================= 2024-03-26T04:05:09.390 INFO:teuthology.orchestra.run.smithi183.stdout:Install 36 Packages 2024-03-26T04:05:09.390 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:05:09.393 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 30 M 2024-03-26T04:05:09.393 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 107 M 2024-03-26T04:05:09.393 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-26T04:05:09.894 INFO:teuthology.orchestra.run.smithi183.stdout:(1/36): ceph-19.0.0-1775.g5e70124b.el9.x86_64.r 41 kB/s | 6.6 kB 00:00 2024-03-26T04:05:10.152 INFO:teuthology.orchestra.run.smithi183.stdout:(2/36): ceph-mds-19.0.0-1775.g5e70124b.el9.x86_ 5.5 MB/s | 2.3 MB 00:00 2024-03-26T04:05:10.186 INFO:teuthology.orchestra.run.smithi183.stdout:(3/36): ceph-mgr-19.0.0-1775.g5e70124b.el9.x86_ 3.3 MB/s | 1.5 MB 00:00 2024-03-26T04:05:10.303 INFO:teuthology.orchestra.run.smithi183.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1775.g5e70 2.1 MB/s | 248 kB 00:00 2024-03-26T04:05:10.413 INFO:teuthology.orchestra.run.smithi183.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2024-03-26T04:05:10.453 INFO:teuthology.orchestra.run.smithi183.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-26T04:05:10.487 INFO:teuthology.orchestra.run.smithi183.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 941 kB/s | 32 kB 00:00 2024-03-26T04:05:10.521 INFO:teuthology.orchestra.run.smithi183.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-26T04:05:10.554 INFO:teuthology.orchestra.run.smithi183.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-26T04:05:10.596 INFO:teuthology.orchestra.run.smithi183.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-26T04:05:10.664 INFO:teuthology.orchestra.run.smithi183.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-26T04:05:10.697 INFO:teuthology.orchestra.run.smithi183.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-26T04:05:10.730 INFO:teuthology.orchestra.run.smithi183.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-26T04:05:10.764 INFO:teuthology.orchestra.run.smithi183.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-26T04:05:10.848 INFO:teuthology.orchestra.run.smithi183.stdout:(15/36): ceph-mon-19.0.0-1775.g5e70124b.el9.x86 4.9 MB/s | 4.7 MB 00:00 2024-03-26T04:05:11.049 INFO:teuthology.orchestra.run.smithi183.stdout:(16/36): ceph-osd-19.0.0-1775.g5e70124b.el9.x86 19 MB/s | 17 MB 00:00 2024-03-26T04:05:11.073 INFO:teuthology.orchestra.run.smithi183.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 404 kB/s | 125 kB 00:00 2024-03-26T04:05:11.107 INFO:teuthology.orchestra.run.smithi183.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 86 kB/s | 22 kB 00:00 2024-03-26T04:05:11.426 INFO:teuthology.orchestra.run.smithi183.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 124 kB/s | 43 kB 00:00 2024-03-26T04:05:11.428 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1775.g5e70124b.el9.x86_64 5/6 2024-03-26T04:05:11.504 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-test-2:19.0.0-1775.g5e70124b.el9.x86_64 6/6 2024-03-26T04:05:11.533 INFO:teuthology.orchestra.run.smithi183.stdout:(20/36): luarocks-3.9.2-1.el9.noarch.rpm 312 kB/s | 151 kB 00:00 2024-03-26T04:05:11.616 INFO:teuthology.orchestra.run.smithi183.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 337 kB/s | 172 kB 00:00 2024-03-26T04:05:11.716 INFO:teuthology.orchestra.run.smithi183.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 58 kB/s | 11 kB 00:00 2024-03-26T04:05:11.883 INFO:teuthology.orchestra.run.smithi183.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 66 kB/s | 18 kB 00:00 2024-03-26T04:05:11.992 INFO:teuthology.orchestra.run.smithi183.stdout:(24/36): python3-cherrypy-18.6.1-2.el9.noarch.r 631 kB/s | 358 kB 00:00 2024-03-26T04:05:12.009 INFO:teuthology.orchestra.run.smithi183.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 79 kB/s | 23 kB 00:00 2024-03-26T04:05:12.168 INFO:teuthology.orchestra.run.smithi183.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 68 kB/s | 19 kB 00:00 2024-03-26T04:05:12.251 INFO:teuthology.orchestra.run.smithi183.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 76 kB/s | 20 kB 00:00 2024-03-26T04:05:12.377 INFO:teuthology.orchestra.run.smithi183.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 126 kB/s | 46 kB 00:00 2024-03-26T04:05:12.510 INFO:teuthology.orchestra.run.smithi183.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 230 kB/s | 79 kB 00:00 2024-03-26T04:05:12.635 INFO:teuthology.orchestra.run.smithi183.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 63 kB/s | 16 kB 00:00 2024-03-26T04:05:12.802 INFO:teuthology.orchestra.run.smithi183.stdout:(31/36): python3-pecan-1.4.2-3.el9.noarch.rpm 494 kB/s | 272 kB 00:00 2024-03-26T04:05:12.869 INFO:teuthology.orchestra.run.smithi183.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 250 kB/s | 90 kB 00:00 2024-03-26T04:05:12.903 INFO:teuthology.orchestra.run.smithi183.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 134 kB/s | 36 kB 00:00 2024-03-26T04:05:13.161 INFO:teuthology.orchestra.run.smithi183.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 77 kB/s | 20 kB 00:00 2024-03-26T04:05:13.328 INFO:teuthology.orchestra.run.smithi183.stdout:(35/36): python3-webob-1.8.7-6.el9.noarch.rpm 437 kB/s | 230 kB 00:00 2024-03-26T04:05:13.453 INFO:teuthology.orchestra.run.smithi183.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 732 kB/s | 427 kB 00:00 2024-03-26T04:05:13.458 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:05:13.458 INFO:teuthology.orchestra.run.smithi183.stdout:Total 7.3 MB/s | 30 MB 00:04 2024-03-26T04:05:13.483 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-26T04:05:13.525 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-26T04:05:13.526 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-26T04:05:13.937 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-26T04:05:13.937 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-26T04:05:14.037 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-test-2:19.0.0-1775.g5e70124b.el9.x86_64 6/6 2024-03-26T04:05:14.037 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-test-2:19.0.0-1775.g5e70124b.el9.x86_64 1/6 2024-03-26T04:05:14.037 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephsqlite-2:19.0.0-1775.g5e70124b.el9.x86_64 2/6 2024-03-26T04:05:14.037 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-26T04:05:14.037 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-26T04:05:14.037 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-26T04:05:14.427 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-26T04:05:14.427 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:05:14.427 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-26T04:05:14.427 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-test-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:05:14.427 INFO:teuthology.orchestra.run.smithi016.stdout: jq-1.6-16.el9.x86_64 2024-03-26T04:05:14.427 INFO:teuthology.orchestra.run.smithi016.stdout: libcephsqlite-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:05:14.428 INFO:teuthology.orchestra.run.smithi016.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-26T04:05:14.428 INFO:teuthology.orchestra.run.smithi016.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-26T04:05:14.428 INFO:teuthology.orchestra.run.smithi016.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-26T04:05:14.428 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:05:14.428 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-26T04:05:14.784 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph 2024-03-26T04:05:15.286 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-26T04:05:15.356 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-26T04:05:15.359 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:00:43 ago on Tue 26 Mar 2024 04:04:32 AM UTC. 2024-03-26T04:05:15.397 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-26T04:05:15.433 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-26T04:05:15.503 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-26T04:05:15.537 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-26T04:05:15.542 INFO:teuthology.orchestra.run.smithi016.stdout:========================================================================================= 2024-03-26T04:05:15.542 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-03-26T04:05:15.542 INFO:teuthology.orchestra.run.smithi016.stdout:========================================================================================= 2024-03-26T04:05:15.542 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-26T04:05:15.542 INFO:teuthology.orchestra.run.smithi016.stdout: ceph x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 6.6 k 2024-03-26T04:05:15.542 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-03-26T04:05:15.543 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mds x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 2.3 M 2024-03-26T04:05:15.543 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 1.5 M 2024-03-26T04:05:15.543 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 248 k 2024-03-26T04:05:15.543 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mon x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 4.7 M 2024-03-26T04:05:15.543 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-osd x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 17 M 2024-03-26T04:05:15.543 INFO:teuthology.orchestra.run.smithi016.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-26T04:05:15.543 INFO:teuthology.orchestra.run.smithi016.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-26T04:05:15.543 INFO:teuthology.orchestra.run.smithi016.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-26T04:05:15.543 INFO:teuthology.orchestra.run.smithi016.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-26T04:05:15.543 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-26T04:05:15.543 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-26T04:05:15.543 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-26T04:05:15.543 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-26T04:05:15.543 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-26T04:05:15.544 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-26T04:05:15.544 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-26T04:05:15.544 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-26T04:05:15.544 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-26T04:05:15.544 INFO:teuthology.orchestra.run.smithi016.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-26T04:05:15.544 INFO:teuthology.orchestra.run.smithi016.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-26T04:05:15.544 INFO:teuthology.orchestra.run.smithi016.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-26T04:05:15.544 INFO:teuthology.orchestra.run.smithi016.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-26T04:05:15.544 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-26T04:05:15.544 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-26T04:05:15.544 INFO:teuthology.orchestra.run.smithi016.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-26T04:05:15.544 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-26T04:05:15.544 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-26T04:05:15.544 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-26T04:05:15.545 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-26T04:05:15.545 INFO:teuthology.orchestra.run.smithi016.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-26T04:05:15.545 INFO:teuthology.orchestra.run.smithi016.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-26T04:05:15.545 INFO:teuthology.orchestra.run.smithi016.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-26T04:05:15.545 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-26T04:05:15.545 INFO:teuthology.orchestra.run.smithi016.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-26T04:05:15.545 INFO:teuthology.orchestra.run.smithi016.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-26T04:05:15.545 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:05:15.545 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-26T04:05:15.545 INFO:teuthology.orchestra.run.smithi016.stdout:========================================================================================= 2024-03-26T04:05:15.545 INFO:teuthology.orchestra.run.smithi016.stdout:Install 36 Packages 2024-03-26T04:05:15.545 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:05:15.546 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 30 M 2024-03-26T04:05:15.546 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 107 M 2024-03-26T04:05:15.546 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-26T04:05:15.565 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-26T04:05:15.598 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-26T04:05:15.688 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-26T04:05:15.776 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-26T04:05:15.818 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-26T04:05:15.892 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-26T04:05:15.944 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-26T04:05:15.991 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-26T04:05:16.039 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-26T04:05:16.097 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-26T04:05:16.288 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-26T04:05:16.345 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-26T04:05:16.389 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-26T04:05:16.435 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-26T04:05:16.465 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-26T04:05:16.530 INFO:teuthology.orchestra.run.smithi016.stdout:(1/36): ceph-19.0.0-1775.g5e70124b.el9.x86_64.r 31 kB/s | 6.6 kB 00:00 2024-03-26T04:05:16.550 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-26T04:05:16.583 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-26T04:05:16.617 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-26T04:05:16.657 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-26T04:05:16.692 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-26T04:05:16.696 INFO:teuthology.orchestra.run.smithi016.stdout:(2/36): ceph-mgr-19.0.0-1775.g5e70124b.el9.x86_ 3.9 MB/s | 1.5 MB 00:00 2024-03-26T04:05:16.727 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-26T04:05:16.816 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-26T04:05:16.850 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-26T04:05:16.922 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-26T04:05:17.005 INFO:teuthology.orchestra.run.smithi016.stdout:(3/36): ceph-mds-19.0.0-1775.g5e70124b.el9.x86_ 3.4 MB/s | 2.3 MB 00:00 2024-03-26T04:05:17.069 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-26T04:05:17.106 INFO:teuthology.orchestra.run.smithi016.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1775.g5e70 2.4 MB/s | 248 kB 00:00 2024-03-26T04:05:17.200 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-26T04:05:17.214 INFO:teuthology.orchestra.run.smithi016.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2024-03-26T04:05:17.256 INFO:teuthology.orchestra.run.smithi016.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-26T04:05:17.290 INFO:teuthology.orchestra.run.smithi016.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 952 kB/s | 32 kB 00:00 2024-03-26T04:05:17.348 INFO:teuthology.orchestra.run.smithi016.stdout:(8/36): ceph-mon-19.0.0-1775.g5e70124b.el9.x86_ 5.7 MB/s | 4.7 MB 00:00 2024-03-26T04:05:17.358 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1775.g5e70124b.el9. 31/36 2024-03-26T04:05:17.374 INFO:teuthology.orchestra.run.smithi016.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 565 kB/s | 47 kB 00:00 2024-03-26T04:05:17.388 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 32/36 2024-03-26T04:05:17.399 INFO:teuthology.orchestra.run.smithi016.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 769 kB/s | 39 kB 00:00 2024-03-26T04:05:17.414 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 32/36 2024-03-26T04:05:17.414 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:05:17.415 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-26T04:05:17.415 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T04:05:17.415 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T04:05:17.415 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:05:17.424 INFO:teuthology.orchestra.run.smithi016.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-26T04:05:17.475 INFO:teuthology.orchestra.run.smithi016.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-26T04:05:17.499 INFO:teuthology.orchestra.run.smithi016.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 103 kB 00:00 2024-03-26T04:05:17.516 INFO:teuthology.orchestra.run.smithi016.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-26T04:05:17.542 INFO:teuthology.orchestra.run.smithi016.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-26T04:05:17.726 INFO:teuthology.orchestra.run.smithi016.stdout:(16/36): ceph-osd-19.0.0-1775.g5e70124b.el9.x86 16 MB/s | 17 MB 00:01 2024-03-26T04:05:17.784 INFO:teuthology.orchestra.run.smithi016.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 92 kB/s | 22 kB 00:00 2024-03-26T04:05:17.809 INFO:teuthology.orchestra.run.smithi016.stdout:(18/36): python3-requests-2.25.1-8.el9.noarch.r 427 kB/s | 125 kB 00:00 2024-03-26T04:05:18.847 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-osd-2:19.0.0-1775.g5e70124b.el9.x86_64 33/36 2024-03-26T04:05:18.874 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-osd-2:19.0.0-1775.g5e70124b.el9.x86_64 33/36 2024-03-26T04:05:18.874 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:05:18.874 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-26T04:05:18.874 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T04:05:18.874 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T04:05:18.874 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:05:19.323 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mon-2:19.0.0-1775.g5e70124b.el9.x86_64 34/36 2024-03-26T04:05:19.346 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mon-2:19.0.0-1775.g5e70124b.el9.x86_64 34/36 2024-03-26T04:05:19.346 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:05:19.346 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-26T04:05:19.346 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T04:05:19.347 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T04:05:19.347 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:05:19.574 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mds-2:19.0.0-1775.g5e70124b.el9.x86_64 35/36 2024-03-26T04:05:19.597 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mds-2:19.0.0-1775.g5e70124b.el9.x86_64 35/36 2024-03-26T04:05:19.597 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:05:19.597 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-26T04:05:19.597 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T04:05:19.597 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T04:05:19.597 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:05:19.662 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-2:19.0.0-1775.g5e70124b.el9.x86_64 36/36 2024-03-26T04:05:21.475 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-2:19.0.0-1775.g5e70124b.el9.x86_64 36/36 2024-03-26T04:05:21.475 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-2:19.0.0-1775.g5e70124b.el9.x86_64 1/36 2024-03-26T04:05:21.475 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mds-2:19.0.0-1775.g5e70124b.el9.x86_64 2/36 2024-03-26T04:05:21.476 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 3/36 2024-03-26T04:05:21.476 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mon-2:19.0.0-1775.g5e70124b.el9.x86_64 4/36 2024-03-26T04:05:21.476 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-osd-2:19.0.0-1775.g5e70124b.el9.x86_64 5/36 2024-03-26T04:05:21.476 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1775.g5e70124b.el9. 6/36 2024-03-26T04:05:21.476 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-26T04:05:21.476 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-26T04:05:21.476 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-26T04:05:21.476 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-26T04:05:21.476 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-26T04:05:21.476 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-26T04:05:21.476 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-26T04:05:21.476 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-26T04:05:21.476 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-26T04:05:21.477 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-26T04:05:21.477 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-26T04:05:21.477 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-26T04:05:21.477 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-26T04:05:21.477 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-26T04:05:21.477 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-26T04:05:21.477 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-26T04:05:21.477 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-26T04:05:21.477 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-26T04:05:21.477 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-26T04:05:21.477 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-26T04:05:21.477 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-26T04:05:21.478 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-26T04:05:21.479 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-26T04:05:21.479 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-26T04:05:21.479 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-26T04:05:21.479 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-26T04:05:21.479 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-26T04:05:21.479 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-26T04:05:21.479 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-26T04:05:22.601 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-26T04:05:22.602 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:05:22.602 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-26T04:05:22.602 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:05:22.602 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mds-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:05:22.602 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:05:22.602 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-modules-core-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:05:22.602 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mon-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:05:22.602 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-osd-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:05:22.602 INFO:teuthology.orchestra.run.smithi183.stdout: lua-5.4.4-4.el9.x86_64 2024-03-26T04:05:22.602 INFO:teuthology.orchestra.run.smithi183.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-26T04:05:22.603 INFO:teuthology.orchestra.run.smithi183.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-26T04:05:22.603 INFO:teuthology.orchestra.run.smithi183.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-26T04:05:22.603 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-26T04:05:22.603 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-26T04:05:22.603 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-26T04:05:22.603 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-26T04:05:22.603 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-26T04:05:22.603 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-26T04:05:22.603 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-26T04:05:22.603 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-26T04:05:22.603 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-26T04:05:22.603 INFO:teuthology.orchestra.run.smithi183.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-26T04:05:22.604 INFO:teuthology.orchestra.run.smithi183.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-26T04:05:22.604 INFO:teuthology.orchestra.run.smithi183.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-26T04:05:22.604 INFO:teuthology.orchestra.run.smithi183.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-26T04:05:22.604 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-26T04:05:22.604 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ply-3.11-14.el9.noarch 2024-03-26T04:05:22.604 INFO:teuthology.orchestra.run.smithi183.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-26T04:05:22.604 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-26T04:05:22.604 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-26T04:05:22.604 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-26T04:05:22.604 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-26T04:05:22.604 INFO:teuthology.orchestra.run.smithi183.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-26T04:05:22.604 INFO:teuthology.orchestra.run.smithi183.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-26T04:05:22.605 INFO:teuthology.orchestra.run.smithi183.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-26T04:05:22.605 INFO:teuthology.orchestra.run.smithi183.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-26T04:05:22.605 INFO:teuthology.orchestra.run.smithi183.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-26T04:05:22.605 INFO:teuthology.orchestra.run.smithi183.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-26T04:05:22.605 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:05:22.605 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-26T04:05:22.957 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-base 2024-03-26T04:05:23.528 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:55 ago on Tue 26 Mar 2024 04:04:28 AM UTC. 2024-03-26T04:05:23.624 INFO:teuthology.orchestra.run.smithi183.stdout:Package ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:05:23.676 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-26T04:05:23.677 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-26T04:05:23.677 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-26T04:05:23.738 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install cephadm 2024-03-26T04:05:24.289 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:56 ago on Tue 26 Mar 2024 04:04:28 AM UTC. 2024-03-26T04:05:24.436 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-26T04:05:24.436 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-26T04:05:24.437 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-26T04:05:24.437 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-26T04:05:24.437 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-26T04:05:24.437 INFO:teuthology.orchestra.run.smithi183.stdout: cephadm noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 748 k 2024-03-26T04:05:24.437 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:05:24.437 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-26T04:05:24.437 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-26T04:05:24.437 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-26T04:05:24.437 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:05:24.437 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 748 k 2024-03-26T04:05:24.437 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 754 k 2024-03-26T04:05:24.437 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-26T04:05:24.875 INFO:teuthology.orchestra.run.smithi183.stdout:cephadm-19.0.0-1775.g5e70124b.el9.noarch.rpm 1.7 MB/s | 748 kB 00:00 2024-03-26T04:05:24.876 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:05:24.876 INFO:teuthology.orchestra.run.smithi183.stdout:Total 1.7 MB/s | 748 kB 00:00 2024-03-26T04:05:24.876 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-26T04:05:24.881 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-26T04:05:24.881 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-26T04:05:24.887 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-26T04:05:24.887 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-26T04:05:24.958 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-26T04:05:25.358 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/1 2024-03-26T04:05:25.429 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/1 2024-03-26T04:05:25.601 INFO:teuthology.orchestra.run.smithi016.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 19 kB/s | 151 kB 00:07 2024-03-26T04:05:25.807 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/1 2024-03-26T04:05:26.051 INFO:teuthology.orchestra.run.smithi016.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 797 kB/s | 358 kB 00:00 2024-03-26T04:05:26.093 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/1 2024-03-26T04:05:26.093 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:05:26.093 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-26T04:05:26.093 INFO:teuthology.orchestra.run.smithi183.stdout: cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:05:26.093 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:05:26.093 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-26T04:05:26.185 INFO:teuthology.orchestra.run.smithi016.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 80 kB/s | 11 kB 00:00 2024-03-26T04:05:26.325 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-immutable-object-cache 2024-03-26T04:05:26.335 INFO:teuthology.orchestra.run.smithi016.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 118 kB/s | 18 kB 00:00 2024-03-26T04:05:26.469 INFO:teuthology.orchestra.run.smithi016.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 174 kB/s | 23 kB 00:00 2024-03-26T04:05:26.602 INFO:teuthology.orchestra.run.smithi016.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 146 kB/s | 19 kB 00:00 2024-03-26T04:05:26.736 INFO:teuthology.orchestra.run.smithi016.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 147 kB/s | 20 kB 00:00 2024-03-26T04:05:26.891 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:58 ago on Tue 26 Mar 2024 04:04:28 AM UTC. 2024-03-26T04:05:26.894 INFO:teuthology.orchestra.run.smithi016.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 293 kB/s | 46 kB 00:00 2024-03-26T04:05:27.040 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-26T04:05:27.041 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-26T04:05:27.041 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-03-26T04:05:27.041 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-26T04:05:27.041 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-26T04:05:27.041 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 145 k 2024-03-26T04:05:27.041 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:05:27.041 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-26T04:05:27.041 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-26T04:05:27.041 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-26T04:05:27.041 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:05:27.041 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 145 k 2024-03-26T04:05:27.042 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 436 k 2024-03-26T04:05:27.042 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-26T04:05:27.045 INFO:teuthology.orchestra.run.smithi016.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 525 kB/s | 79 kB 00:00 2024-03-26T04:05:27.265 INFO:teuthology.orchestra.run.smithi183.stdout:ceph-immutable-object-cache-19.0.0-1775.g5e7012 650 kB/s | 145 kB 00:00 2024-03-26T04:05:27.265 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:05:27.266 INFO:teuthology.orchestra.run.smithi183.stdout:Total 647 kB/s | 145 kB 00:00 2024-03-26T04:05:27.266 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-26T04:05:27.273 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-26T04:05:27.273 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-26T04:05:27.325 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-26T04:05:27.325 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-26T04:05:27.387 INFO:teuthology.orchestra.run.smithi016.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 797 kB/s | 272 kB 00:00 2024-03-26T04:05:27.463 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-26T04:05:27.516 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1775.g5e70124b. 1/1 2024-03-26T04:05:27.529 INFO:teuthology.orchestra.run.smithi016.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 116 kB/s | 16 kB 00:00 2024-03-26T04:05:27.540 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1775.g5e70124b. 1/1 2024-03-26T04:05:27.540 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:05:27.540 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-26T04:05:27.540 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:05:27.662 INFO:teuthology.orchestra.run.smithi016.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 675 kB/s | 90 kB 00:00 2024-03-26T04:05:27.829 INFO:teuthology.orchestra.run.smithi016.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 214 kB/s | 36 kB 00:00 2024-03-26T04:05:28.304 INFO:teuthology.orchestra.run.smithi016.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 484 kB/s | 230 kB 00:00 2024-03-26T04:05:28.534 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1775.g5e70124b. 1/1 2024-03-26T04:05:28.534 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:05:28.534 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-26T04:05:28.534 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-immutable-object-cache-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:05:28.534 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:05:28.534 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-26T04:05:28.772 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr 2024-03-26T04:05:29.196 INFO:teuthology.orchestra.run.smithi016.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 479 kB/s | 427 kB 00:00 2024-03-26T04:05:29.330 INFO:teuthology.orchestra.run.smithi016.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 150 kB/s | 20 kB 00:00 2024-03-26T04:05:29.341 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:01 ago on Tue 26 Mar 2024 04:04:28 AM UTC. 2024-03-26T04:05:29.440 INFO:teuthology.orchestra.run.smithi183.stdout:Package ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:05:29.492 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-26T04:05:29.493 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-26T04:05:29.493 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-26T04:05:29.553 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr-dashboard 2024-03-26T04:05:30.104 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:02 ago on Tue 26 Mar 2024 04:04:28 AM UTC. 2024-03-26T04:05:30.253 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-26T04:05:30.255 INFO:teuthology.orchestra.run.smithi183.stdout:====================================================================================== 2024-03-26T04:05:30.255 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-26T04:05:30.255 INFO:teuthology.orchestra.run.smithi183.stdout:====================================================================================== 2024-03-26T04:05:30.255 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-26T04:05:30.255 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 1.8 M 2024-03-26T04:05:30.255 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-26T04:05:30.255 INFO:teuthology.orchestra.run.smithi183.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-26T04:05:30.255 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 31 k 2024-03-26T04:05:30.255 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 15 k 2024-03-26T04:05:30.255 INFO:teuthology.orchestra.run.smithi183.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-26T04:05:30.255 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-26T04:05:30.255 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-26T04:05:30.256 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-26T04:05:30.256 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-26T04:05:30.256 INFO:teuthology.orchestra.run.smithi183.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-26T04:05:30.256 INFO:teuthology.orchestra.run.smithi183.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-26T04:05:30.256 INFO:teuthology.orchestra.run.smithi183.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-26T04:05:30.256 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:05:30.256 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-26T04:05:30.256 INFO:teuthology.orchestra.run.smithi183.stdout:====================================================================================== 2024-03-26T04:05:30.256 INFO:teuthology.orchestra.run.smithi183.stdout:Install 12 Packages 2024-03-26T04:05:30.256 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:05:30.257 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 6.9 M 2024-03-26T04:05:30.257 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 26 M 2024-03-26T04:05:30.257 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-26T04:05:31.119 INFO:teuthology.orchestra.run.smithi183.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1775.g5e 160 kB/s | 31 kB 00:00 2024-03-26T04:05:31.244 INFO:teuthology.orchestra.run.smithi183.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1775.g5e7 48 kB/s | 15 kB 00:00 2024-03-26T04:05:31.294 INFO:teuthology.orchestra.run.smithi183.stdout:(3/12): ceph-mgr-dashboard-19.0.0-1775.g5e70124 4.8 MB/s | 1.8 MB 00:00 2024-03-26T04:05:31.403 INFO:teuthology.orchestra.run.smithi183.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.6 MB/s | 1.0 MB 00:00 2024-03-26T04:05:31.429 INFO:teuthology.orchestra.run.smithi183.stdout:(5/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.3 MB/s | 238 kB 00:00 2024-03-26T04:05:31.604 INFO:teuthology.orchestra.run.smithi183.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 111 kB/s | 19 kB 00:00 2024-03-26T04:05:31.771 INFO:teuthology.orchestra.run.smithi183.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.5 MB/s | 548 kB 00:00 2024-03-26T04:05:31.837 INFO:teuthology.orchestra.run.smithi183.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.1 MB/s | 144 kB 00:00 2024-03-26T04:05:31.879 INFO:teuthology.orchestra.run.smithi183.stdout:(9/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.4 MB/s | 863 kB 00:00 2024-03-26T04:05:31.905 INFO:teuthology.orchestra.run.smithi183.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 458 kB/s | 31 kB 00:00 2024-03-26T04:05:31.947 INFO:teuthology.orchestra.run.smithi183.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 5.9 MB/s | 2.0 MB 00:00 2024-03-26T04:05:32.138 INFO:teuthology.orchestra.run.smithi183.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 729 kB/s | 188 kB 00:00 2024-03-26T04:05:32.141 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:05:32.141 INFO:teuthology.orchestra.run.smithi183.stdout:Total 3.6 MB/s | 6.9 MB 00:01 2024-03-26T04:05:32.160 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-26T04:05:32.178 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-26T04:05:32.178 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-26T04:05:32.356 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-26T04:05:32.357 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-26T04:05:32.865 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-26T04:05:32.939 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-26T04:05:32.975 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-26T04:05:33.021 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-26T04:05:33.051 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-26T04:05:33.158 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-26T04:05:33.276 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-26T04:05:33.370 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-26T04:05:33.414 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-26T04:05:33.443 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-26T04:05:33.474 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1775.g5e70124b.el9 10/12 2024-03-26T04:05:33.717 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1775.g5e70124b.el 11/12 2024-03-26T04:05:33.749 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1775.g5e70124b.el9.noa 12/12 2024-03-26T04:05:34.555 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1775.g5e70124b.el9.noa 12/12 2024-03-26T04:05:34.555 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1775.g5e70124b.el 1/12 2024-03-26T04:05:34.555 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1775.g5e70124b.el9.noa 2/12 2024-03-26T04:05:34.555 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1775.g5e70124b.el9 3/12 2024-03-26T04:05:34.555 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-26T04:05:34.555 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-26T04:05:34.555 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-26T04:05:34.555 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-26T04:05:34.555 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-26T04:05:34.555 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-26T04:05:34.555 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-26T04:05:34.555 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-26T04:05:34.916 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-26T04:05:34.916 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:05:34.916 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-26T04:05:34.916 INFO:teuthology.orchestra.run.smithi183.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-26T04:05:34.916 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-grafana-dashboards-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:05:34.916 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-dashboard-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:05:34.916 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-prometheus-alerts-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:05:34.916 INFO:teuthology.orchestra.run.smithi183.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-26T04:05:34.916 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-26T04:05:34.916 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-26T04:05:34.917 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-26T04:05:34.917 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-26T04:05:34.917 INFO:teuthology.orchestra.run.smithi183.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-26T04:05:34.917 INFO:teuthology.orchestra.run.smithi183.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-26T04:05:34.917 INFO:teuthology.orchestra.run.smithi183.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-26T04:05:34.917 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:05:34.917 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-26T04:05:35.148 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-26T04:05:35.705 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:07 ago on Tue 26 Mar 2024 04:04:28 AM UTC. 2024-03-26T04:05:35.857 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-26T04:05:35.857 INFO:teuthology.orchestra.run.smithi183.stdout:============================================================================================ 2024-03-26T04:05:35.857 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-26T04:05:35.857 INFO:teuthology.orchestra.run.smithi183.stdout:============================================================================================ 2024-03-26T04:05:35.857 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-26T04:05:35.857 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 7.4 M 2024-03-26T04:05:35.857 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-26T04:05:35.857 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-26T04:05:35.857 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-26T04:05:35.857 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-26T04:05:35.857 INFO:teuthology.orchestra.run.smithi183.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-26T04:05:35.857 INFO:teuthology.orchestra.run.smithi183.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-26T04:05:35.858 INFO:teuthology.orchestra.run.smithi183.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-26T04:05:35.858 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-26T04:05:35.858 INFO:teuthology.orchestra.run.smithi183.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-26T04:05:35.858 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-26T04:05:35.858 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-26T04:05:35.858 INFO:teuthology.orchestra.run.smithi183.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-26T04:05:35.858 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:05:35.858 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-26T04:05:35.858 INFO:teuthology.orchestra.run.smithi183.stdout:============================================================================================ 2024-03-26T04:05:35.858 INFO:teuthology.orchestra.run.smithi183.stdout:Install 12 Packages 2024-03-26T04:05:35.858 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:05:35.859 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 37 M 2024-03-26T04:05:35.859 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 209 M 2024-03-26T04:05:35.859 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-26T04:05:36.262 INFO:teuthology.orchestra.run.smithi183.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 250 kB/s | 32 kB 00:00 2024-03-26T04:05:36.295 INFO:teuthology.orchestra.run.smithi183.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 534 kB/s | 17 kB 00:00 2024-03-26T04:05:36.329 INFO:teuthology.orchestra.run.smithi183.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-26T04:05:36.488 INFO:teuthology.orchestra.run.smithi183.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 30 MB/s | 4.7 MB 00:00 2024-03-26T04:05:36.546 INFO:teuthology.orchestra.run.smithi183.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 7.3 MB/s | 3.0 MB 00:00 2024-03-26T04:05:36.638 INFO:teuthology.orchestra.run.smithi183.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 4.8 MB 00:00 2024-03-26T04:05:36.663 INFO:teuthology.orchestra.run.smithi183.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.9 MB/s | 217 kB 00:00 2024-03-26T04:05:36.839 INFO:teuthology.orchestra.run.smithi183.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.5 MB/s | 803 kB 00:00 2024-03-26T04:05:36.989 INFO:teuthology.orchestra.run.smithi183.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 45 MB/s | 16 MB 00:00 2024-03-26T04:05:37.014 INFO:teuthology.orchestra.run.smithi183.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-03-26T04:05:37.173 INFO:teuthology.orchestra.run.smithi183.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.3 MB/s | 245 kB 00:00 2024-03-26T04:05:37.473 INFO:teuthology.orchestra.run.smithi183.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 5.5 MB/s | 7.4 MB 00:01 2024-03-26T04:05:37.475 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:05:37.475 INFO:teuthology.orchestra.run.smithi183.stdout:Total 23 MB/s | 37 MB 00:01 2024-03-26T04:05:37.515 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-26T04:05:37.539 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-26T04:05:37.539 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-26T04:05:37.893 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-26T04:05:37.893 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-26T04:05:38.599 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-26T04:05:38.662 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-26T04:05:38.702 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-26T04:05:38.804 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-26T04:05:38.847 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-26T04:05:39.028 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-26T04:05:39.079 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-26T04:05:39.165 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-26T04:05:39.814 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-26T04:05:39.926 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-26T04:05:41.181 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-26T04:05:42.217 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-26T04:05:42.246 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1775.g5e701 12/12 2024-03-26T04:05:44.919 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1775.g5e701 12/12 2024-03-26T04:05:44.920 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1775.g5e701 1/12 2024-03-26T04:05:44.920 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-26T04:05:44.920 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-26T04:05:44.920 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-26T04:05:44.920 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-26T04:05:44.920 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-26T04:05:44.920 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-26T04:05:44.920 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-26T04:05:44.920 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-26T04:05:44.920 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-26T04:05:44.920 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-26T04:05:45.386 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-26T04:05:45.386 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:05:45.387 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-26T04:05:45.387 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:05:45.387 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-26T04:05:45.387 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-26T04:05:45.387 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-26T04:05:45.387 INFO:teuthology.orchestra.run.smithi183.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-26T04:05:45.387 INFO:teuthology.orchestra.run.smithi183.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-26T04:05:45.387 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-26T04:05:45.387 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-26T04:05:45.387 INFO:teuthology.orchestra.run.smithi183.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-26T04:05:45.387 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-26T04:05:45.387 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-26T04:05:45.387 INFO:teuthology.orchestra.run.smithi183.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-26T04:05:45.387 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:05:45.388 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-26T04:05:45.718 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr-rook 2024-03-26T04:05:46.289 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:18 ago on Tue 26 Mar 2024 04:04:28 AM UTC. 2024-03-26T04:05:46.438 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-26T04:05:46.439 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2024-03-26T04:05:46.439 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-26T04:05:46.439 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2024-03-26T04:05:46.439 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-26T04:05:46.439 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-rook noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 49 k 2024-03-26T04:05:46.439 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-26T04:05:46.439 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-26T04:05:46.440 INFO:teuthology.orchestra.run.smithi183.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-26T04:05:46.440 INFO:teuthology.orchestra.run.smithi183.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-26T04:05:46.440 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-26T04:05:46.440 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-26T04:05:46.440 INFO:teuthology.orchestra.run.smithi183.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-26T04:05:46.440 INFO:teuthology.orchestra.run.smithi183.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-26T04:05:46.440 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-26T04:05:46.440 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-26T04:05:46.440 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-26T04:05:46.440 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-26T04:05:46.440 INFO:teuthology.orchestra.run.smithi183.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-26T04:05:46.440 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:05:46.440 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-26T04:05:46.440 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2024-03-26T04:05:46.440 INFO:teuthology.orchestra.run.smithi183.stdout:Install 13 Packages 2024-03-26T04:05:46.441 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:05:46.441 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 2.0 M 2024-03-26T04:05:46.441 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 26 M 2024-03-26T04:05:46.441 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-26T04:05:46.931 INFO:teuthology.orchestra.run.smithi183.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 189 kB/s | 24 kB 00:00 2024-03-26T04:05:46.956 INFO:teuthology.orchestra.run.smithi183.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 117 kB/s | 18 kB 00:00 2024-03-26T04:05:47.014 INFO:teuthology.orchestra.run.smithi183.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.1 MB/s | 174 kB 00:00 2024-03-26T04:05:47.040 INFO:teuthology.orchestra.run.smithi183.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-03-26T04:05:47.065 INFO:teuthology.orchestra.run.smithi183.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-26T04:05:47.090 INFO:teuthology.orchestra.run.smithi183.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 853 kB/s | 43 kB 00:00 2024-03-26T04:05:47.116 INFO:teuthology.orchestra.run.smithi183.stdout:(7/13): ceph-mgr-rook-19.0.0-1775.g5e70124b.el9 157 kB/s | 49 kB 00:00 2024-03-26T04:05:47.283 INFO:teuthology.orchestra.run.smithi183.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 78 kB/s | 15 kB 00:00 2024-03-26T04:05:47.308 INFO:teuthology.orchestra.run.smithi183.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 133 kB/s | 32 kB 00:00 2024-03-26T04:05:47.442 INFO:teuthology.orchestra.run.smithi183.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 443 kB/s | 59 kB 00:00 2024-03-26T04:05:47.475 INFO:teuthology.orchestra.run.smithi183.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 564 kB/s | 203 kB 00:00 2024-03-26T04:05:47.575 INFO:teuthology.orchestra.run.smithi183.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 672 kB/s | 90 kB 00:00 2024-03-26T04:05:47.742 INFO:teuthology.orchestra.run.smithi183.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.2 MB/s | 1.0 MB 00:00 2024-03-26T04:05:47.744 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:05:47.744 INFO:teuthology.orchestra.run.smithi183.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2024-03-26T04:05:47.745 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-26T04:05:47.759 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-26T04:05:47.759 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-26T04:05:47.854 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-26T04:05:47.854 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-26T04:05:48.377 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-26T04:05:48.549 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-26T04:05:48.610 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-26T04:05:48.666 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-26T04:05:48.703 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-26T04:05:48.752 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-26T04:05:48.854 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-26T04:05:48.953 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-26T04:05:49.003 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-26T04:05:49.724 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-26T04:05:49.778 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-26T04:05:49.812 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-26T04:05:49.859 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-26T04:05:49.881 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-rook-2:19.0.0-1775.g5e70124b.el9.noarch 13/13 2024-03-26T04:05:50.195 INFO:teuthology.orchestra.run.smithi016.stdout:(35/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.3 kB/s | 43 kB 00:32 2024-03-26T04:05:50.630 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1775.g5e70124b.el9.noarch 13/13 2024-03-26T04:05:50.630 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1775.g5e70124b.el9.noarch 1/13 2024-03-26T04:05:50.630 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-26T04:05:50.630 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-26T04:05:50.630 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-26T04:05:50.630 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-26T04:05:50.630 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-26T04:05:50.630 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-26T04:05:50.630 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-26T04:05:50.630 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-26T04:05:50.631 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-26T04:05:50.631 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-26T04:05:50.631 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-26T04:05:50.999 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-26T04:05:50.999 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:05:50.999 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-26T04:05:50.999 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-rook-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:05:50.999 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-26T04:05:50.999 INFO:teuthology.orchestra.run.smithi183.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-26T04:05:50.999 INFO:teuthology.orchestra.run.smithi183.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-26T04:05:50.999 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-26T04:05:50.999 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-26T04:05:50.999 INFO:teuthology.orchestra.run.smithi183.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-26T04:05:50.999 INFO:teuthology.orchestra.run.smithi183.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-26T04:05:51.000 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-26T04:05:51.000 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-26T04:05:51.000 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-26T04:05:51.000 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-26T04:05:51.000 INFO:teuthology.orchestra.run.smithi183.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-26T04:05:51.000 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:05:51.000 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-26T04:05:51.218 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr-cephadm 2024-03-26T04:05:51.777 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:23 ago on Tue 26 Mar 2024 04:04:28 AM UTC. 2024-03-26T04:05:51.926 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-26T04:05:51.926 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2024-03-26T04:05:51.927 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-26T04:05:51.927 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2024-03-26T04:05:51.927 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-26T04:05:51.927 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 138 k 2024-03-26T04:05:51.927 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-26T04:05:51.927 INFO:teuthology.orchestra.run.smithi183.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-26T04:05:51.927 INFO:teuthology.orchestra.run.smithi183.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-26T04:05:51.927 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-26T04:05:51.927 INFO:teuthology.orchestra.run.smithi183.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-26T04:05:51.927 INFO:teuthology.orchestra.run.smithi183.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-26T04:05:51.927 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:05:51.927 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-26T04:05:51.927 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2024-03-26T04:05:51.928 INFO:teuthology.orchestra.run.smithi183.stdout:Install 6 Packages 2024-03-26T04:05:51.928 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:05:51.928 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 6.8 M 2024-03-26T04:05:51.928 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 33 M 2024-03-26T04:05:51.928 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-26T04:05:52.254 INFO:teuthology.orchestra.run.smithi183.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.9 MB/s | 248 kB 00:00 2024-03-26T04:05:52.413 INFO:teuthology.orchestra.run.smithi183.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2024-03-26T04:05:52.446 INFO:teuthology.orchestra.run.smithi183.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1775.g5e70124b.e 435 kB/s | 138 kB 00:00 2024-03-26T04:05:56.338 INFO:teuthology.orchestra.run.smithi183.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 134 kB/s | 548 kB 00:04 2024-03-26T04:06:00.263 INFO:teuthology.orchestra.run.smithi183.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 6.5 kB/s | 51 kB 00:07 2024-03-26T04:06:24.526 INFO:teuthology.orchestra.run.smithi183.stdout:(6/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.8 kB/s | 58 kB 00:32 2024-03-26T04:06:24.527 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:06:24.527 INFO:teuthology.orchestra.run.smithi183.stdout:Total 213 kB/s | 6.8 MB 00:32 2024-03-26T04:06:24.541 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-26T04:06:24.554 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-26T04:06:24.554 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-26T04:06:24.647 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-26T04:06:24.647 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-26T04:06:24.950 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-26T04:06:25.124 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-26T04:06:25.185 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-26T04:06:25.703 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-26T04:06:25.804 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-26T04:06:25.867 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-26T04:06:25.889 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 6/6 2024-03-26T04:06:26.652 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 6/6 2024-03-26T04:06:26.653 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/6 2024-03-26T04:06:26.653 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-26T04:06:26.653 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-26T04:06:26.653 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-26T04:06:26.653 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-26T04:06:26.958 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-26T04:06:26.958 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:06:26.958 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-26T04:06:26.959 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:06:26.959 INFO:teuthology.orchestra.run.smithi183.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-26T04:06:26.959 INFO:teuthology.orchestra.run.smithi183.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-26T04:06:26.959 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-26T04:06:26.959 INFO:teuthology.orchestra.run.smithi183.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-26T04:06:26.959 INFO:teuthology.orchestra.run.smithi183.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-26T04:06:26.959 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:06:26.959 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-26T04:06:27.243 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-fuse 2024-03-26T04:06:27.804 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:59 ago on Tue 26 Mar 2024 04:04:28 AM UTC. 2024-03-26T04:06:27.953 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-26T04:06:27.953 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-26T04:06:27.953 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-26T04:06:27.953 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-26T04:06:27.954 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-26T04:06:27.954 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-fuse x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 888 k 2024-03-26T04:06:27.954 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-26T04:06:27.954 INFO:teuthology.orchestra.run.smithi183.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-26T04:06:27.954 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:06:27.954 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-26T04:06:27.954 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-26T04:06:27.954 INFO:teuthology.orchestra.run.smithi183.stdout:Install 2 Packages 2024-03-26T04:06:27.954 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:06:27.955 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 967 k 2024-03-26T04:06:27.955 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 2.8 M 2024-03-26T04:06:27.955 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-26T04:06:28.111 INFO:teuthology.orchestra.run.smithi183.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 504 kB/s | 79 kB 00:00 2024-03-26T04:06:28.411 INFO:teuthology.orchestra.run.smithi183.stdout:(2/2): ceph-fuse-19.0.0-1775.g5e70124b.el9.x86_ 1.9 MB/s | 888 kB 00:00 2024-03-26T04:06:28.412 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:06:28.412 INFO:teuthology.orchestra.run.smithi183.stdout:Total 2.1 MB/s | 967 kB 00:00 2024-03-26T04:06:28.412 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-26T04:06:28.433 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-26T04:06:28.433 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-26T04:06:28.506 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-26T04:06:28.506 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-26T04:06:28.696 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-26T04:06:28.838 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-26T04:06:28.920 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 2/2 2024-03-26T04:06:29.618 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 2/2 2024-03-26T04:06:29.618 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 1/2 2024-03-26T04:06:29.949 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-26T04:06:29.949 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:06:29.949 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-26T04:06:29.949 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-26T04:06:29.949 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:06:29.949 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-26T04:06:30.216 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-volume 2024-03-26T04:06:30.791 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:02 ago on Tue 26 Mar 2024 04:04:28 AM UTC. 2024-03-26T04:06:30.945 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-26T04:06:30.945 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-26T04:06:30.945 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-26T04:06:30.946 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-26T04:06:30.946 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-26T04:06:30.946 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-volume noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 264 k 2024-03-26T04:06:30.946 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-26T04:06:30.946 INFO:teuthology.orchestra.run.smithi183.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-26T04:06:30.946 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:06:30.946 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-26T04:06:30.946 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-26T04:06:30.946 INFO:teuthology.orchestra.run.smithi183.stdout:Install 2 Packages 2024-03-26T04:06:30.946 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:06:30.947 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 332 k 2024-03-26T04:06:30.947 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 1.5 M 2024-03-26T04:06:30.947 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-26T04:06:31.066 INFO:teuthology.orchestra.run.smithi183.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 578 kB/s | 69 kB 00:00 2024-03-26T04:06:31.216 INFO:teuthology.orchestra.run.smithi183.stdout:(2/2): ceph-volume-19.0.0-1775.g5e70124b.el9.no 979 kB/s | 264 kB 00:00 2024-03-26T04:06:31.218 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:06:31.218 INFO:teuthology.orchestra.run.smithi183.stdout:Total 1.2 MB/s | 332 kB 00:00 2024-03-26T04:06:31.218 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-26T04:06:31.226 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-26T04:06:31.227 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-26T04:06:31.267 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-26T04:06:31.267 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-26T04:06:31.414 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-26T04:06:31.597 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-26T04:06:31.635 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-volume-2:19.0.0-1775.g5e70124b.el9.noarch 2/2 2024-03-26T04:06:31.651 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-volume-2:19.0.0-1775.g5e70124b.el9.noarch 2/2 2024-03-26T04:06:31.651 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:06:31.651 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-26T04:06:31.651 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:06:32.363 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-volume-2:19.0.0-1775.g5e70124b.el9.noarch 1/2 2024-03-26T04:06:32.714 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-26T04:06:32.714 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:06:32.715 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-26T04:06:32.715 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-volume-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:06:32.715 INFO:teuthology.orchestra.run.smithi183.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-26T04:06:32.715 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:06:32.715 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-26T04:06:33.074 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install librados-devel 2024-03-26T04:06:33.647 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:05 ago on Tue 26 Mar 2024 04:04:28 AM UTC. 2024-03-26T04:06:33.798 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-26T04:06:33.799 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-26T04:06:33.799 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-03-26T04:06:33.799 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-26T04:06:33.799 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-26T04:06:33.799 INFO:teuthology.orchestra.run.smithi183.stdout: librados-devel x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 127 k 2024-03-26T04:06:33.799 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:06:33.799 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-26T04:06:33.799 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-26T04:06:33.799 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-26T04:06:33.799 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:06:33.799 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 127 k 2024-03-26T04:06:33.800 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 456 k 2024-03-26T04:06:33.800 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-26T04:06:34.023 INFO:teuthology.orchestra.run.smithi183.stdout:librados-devel-19.0.0-1775.g5e70124b.el9.x86_64 570 kB/s | 127 kB 00:00 2024-03-26T04:06:34.023 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:06:34.023 INFO:teuthology.orchestra.run.smithi183.stdout:Total 566 kB/s | 127 kB 00:00 2024-03-26T04:06:34.023 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-26T04:06:34.029 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-26T04:06:34.029 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-26T04:06:34.084 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-26T04:06:34.084 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-26T04:06:34.226 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-26T04:06:34.347 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librados-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:06:34.780 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:06:35.089 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:06:35.090 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:06:35.090 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-26T04:06:35.090 INFO:teuthology.orchestra.run.smithi183.stdout: librados-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:06:35.090 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:06:35.090 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-26T04:06:35.309 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install libcephfs2 2024-03-26T04:06:35.880 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:07 ago on Tue 26 Mar 2024 04:04:28 AM UTC. 2024-03-26T04:06:35.981 INFO:teuthology.orchestra.run.smithi183.stdout:Package libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:06:36.033 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-26T04:06:36.034 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-26T04:06:36.035 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-26T04:06:36.110 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install libcephfs-devel 2024-03-26T04:06:36.666 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:08 ago on Tue 26 Mar 2024 04:04:28 AM UTC. 2024-03-26T04:06:36.818 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-26T04:06:36.818 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-26T04:06:36.818 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-03-26T04:06:36.819 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-26T04:06:36.819 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-26T04:06:36.819 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs-devel x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 31 k 2024-03-26T04:06:36.819 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:06:36.819 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-26T04:06:36.819 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-26T04:06:36.819 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-26T04:06:36.819 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:06:36.819 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 31 k 2024-03-26T04:06:36.819 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 140 k 2024-03-26T04:06:36.819 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-26T04:06:37.018 INFO:teuthology.orchestra.run.smithi183.stdout:libcephfs-devel-19.0.0-1775.g5e70124b.el9.x86_6 158 kB/s | 31 kB 00:00 2024-03-26T04:06:37.019 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:06:37.019 INFO:teuthology.orchestra.run.smithi183.stdout:Total 157 kB/s | 31 kB 00:00 2024-03-26T04:06:37.019 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-26T04:06:37.022 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-26T04:06:37.022 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-26T04:06:37.048 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-26T04:06:37.048 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-26T04:06:37.173 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-26T04:06:37.274 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephfs-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:06:37.659 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:06:37.929 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephfs-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:06:37.929 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:06:37.929 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-26T04:06:37.929 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:06:37.929 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:06:37.929 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-26T04:06:38.094 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install librados2 2024-03-26T04:06:38.661 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:10 ago on Tue 26 Mar 2024 04:04:28 AM UTC. 2024-03-26T04:06:38.756 INFO:teuthology.orchestra.run.smithi183.stdout:Package librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:06:38.807 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-26T04:06:38.808 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-26T04:06:38.809 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-26T04:06:38.870 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install librbd1 2024-03-26T04:06:39.428 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:11 ago on Tue 26 Mar 2024 04:04:28 AM UTC. 2024-03-26T04:06:39.524 INFO:teuthology.orchestra.run.smithi183.stdout:Package librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:06:39.575 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-26T04:06:39.576 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-26T04:06:39.576 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-26T04:06:39.620 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-rados 2024-03-26T04:06:40.184 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:12 ago on Tue 26 Mar 2024 04:04:28 AM UTC. 2024-03-26T04:06:40.280 INFO:teuthology.orchestra.run.smithi183.stdout:Package python3-rados-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:06:40.334 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-26T04:06:40.335 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-26T04:06:40.335 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-26T04:06:40.379 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-rgw 2024-03-26T04:06:40.940 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:12 ago on Tue 26 Mar 2024 04:04:28 AM UTC. 2024-03-26T04:06:41.038 INFO:teuthology.orchestra.run.smithi183.stdout:Package python3-rgw-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:06:41.088 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-26T04:06:41.090 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-26T04:06:41.090 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-26T04:06:41.148 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-cephfs 2024-03-26T04:06:41.706 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:13 ago on Tue 26 Mar 2024 04:04:28 AM UTC. 2024-03-26T04:06:41.804 INFO:teuthology.orchestra.run.smithi183.stdout:Package python3-cephfs-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:06:41.855 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-26T04:06:41.856 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-26T04:06:41.856 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-26T04:06:41.909 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-rbd 2024-03-26T04:06:42.469 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:14 ago on Tue 26 Mar 2024 04:04:28 AM UTC. 2024-03-26T04:06:42.566 INFO:teuthology.orchestra.run.smithi183.stdout:Package python3-rbd-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:06:42.620 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-26T04:06:42.621 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-26T04:06:42.621 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-26T04:06:42.676 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install rbd-fuse 2024-03-26T04:06:43.238 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:15 ago on Tue 26 Mar 2024 04:04:28 AM UTC. 2024-03-26T04:06:43.389 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-26T04:06:43.389 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-26T04:06:43.389 INFO:teuthology.orchestra.run.smithi183.stdout: Package Architecture Version Repository Size 2024-03-26T04:06:43.389 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-26T04:06:43.390 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-26T04:06:43.390 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-fuse x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 85 k 2024-03-26T04:06:43.390 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:06:43.390 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-26T04:06:43.390 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-26T04:06:43.390 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-26T04:06:43.390 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:06:43.390 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 85 k 2024-03-26T04:06:43.390 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 231 k 2024-03-26T04:06:43.390 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-26T04:06:43.604 INFO:teuthology.orchestra.run.smithi183.stdout:rbd-fuse-19.0.0-1775.g5e70124b.el9.x86_64.rpm 398 kB/s | 85 kB 00:00 2024-03-26T04:06:43.605 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:06:43.605 INFO:teuthology.orchestra.run.smithi183.stdout:Total 396 kB/s | 85 kB 00:00 2024-03-26T04:06:43.605 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-26T04:06:43.612 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-26T04:06:43.612 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-26T04:06:43.669 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-26T04:06:43.669 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-26T04:06:43.817 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-26T04:06:43.919 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:06:44.325 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:06:44.636 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:06:44.637 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:06:44.637 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-26T04:06:44.637 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:06:44.637 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:06:44.637 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-26T04:06:44.847 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install rbd-mirror 2024-03-26T04:06:45.418 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:17 ago on Tue 26 Mar 2024 04:04:28 AM UTC. 2024-03-26T04:06:45.569 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-26T04:06:45.570 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-26T04:06:45.570 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-03-26T04:06:45.570 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-26T04:06:45.570 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-26T04:06:45.571 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-mirror x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 3.1 M 2024-03-26T04:06:45.571 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:06:45.571 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-26T04:06:45.571 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-26T04:06:45.571 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-26T04:06:45.571 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:06:45.571 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 3.1 M 2024-03-26T04:06:45.571 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 13 M 2024-03-26T04:06:45.571 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-26T04:06:45.949 INFO:teuthology.orchestra.run.smithi183.stdout:rbd-mirror-19.0.0-1775.g5e70124b.el9.x86_64.rpm 8.2 MB/s | 3.1 MB 00:00 2024-03-26T04:06:45.949 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:06:45.950 INFO:teuthology.orchestra.run.smithi183.stdout:Total 8.1 MB/s | 3.1 MB 00:00 2024-03-26T04:06:45.950 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-26T04:06:45.959 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-26T04:06:45.959 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-26T04:06:46.032 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-26T04:06:46.032 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-26T04:06:46.447 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-26T04:06:46.516 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-mirror-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:06:46.540 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:06:46.540 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:06:46.540 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-26T04:06:46.540 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T04:06:46.541 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T04:06:46.541 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:06:47.685 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-mirror-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:06:47.685 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:06:47.685 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-26T04:06:47.685 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-mirror-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:06:47.685 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:06:47.685 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-26T04:06:47.927 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install rbd-nbd 2024-03-26T04:06:48.506 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:20 ago on Tue 26 Mar 2024 04:04:28 AM UTC. 2024-03-26T04:06:48.654 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-26T04:06:48.655 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-26T04:06:48.655 INFO:teuthology.orchestra.run.smithi183.stdout: Package Architecture Version Repository Size 2024-03-26T04:06:48.655 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-26T04:06:48.655 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-26T04:06:48.655 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-nbd x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 171 k 2024-03-26T04:06:48.655 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:06:48.655 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-26T04:06:48.655 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-26T04:06:48.655 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-26T04:06:48.656 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:06:48.656 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 171 k 2024-03-26T04:06:48.656 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 494 k 2024-03-26T04:06:48.656 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-26T04:06:49.194 INFO:teuthology.orchestra.run.smithi183.stdout:rbd-nbd-19.0.0-1775.g5e70124b.el9.x86_64.rpm 318 kB/s | 171 kB 00:00 2024-03-26T04:06:49.195 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:06:49.195 INFO:teuthology.orchestra.run.smithi183.stdout:Total 317 kB/s | 171 kB 00:00 2024-03-26T04:06:49.195 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-26T04:06:49.203 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-26T04:06:49.203 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-26T04:06:49.259 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-26T04:06:49.259 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-26T04:06:49.417 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-26T04:06:49.534 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-nbd-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:06:49.987 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:06:50.286 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-nbd-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:06:50.287 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:06:50.287 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-26T04:06:50.287 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-nbd-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:06:50.287 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:06:50.287 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-26T04:06:50.520 DEBUG:teuthology.parallel:result is None 2024-03-26T04:07:29.965 INFO:teuthology.orchestra.run.smithi016.stdout:[MIRROR] python3-cheroot-8.6.0-4.el9.noarch.rpm: Curl error (28): Timeout was reached for http://epel.stl.us.ssimn.org/9/Everything/x86_64/Packages/p/python3-cheroot-8.6.0-4.el9.noarch.rpm [Failed to connect to epel.stl.us.ssimn.org port 80: Connection timed out] 2024-03-26T04:09:41.037 INFO:teuthology.orchestra.run.smithi016.stdout:[MIRROR] python3-cheroot-8.6.0-4.el9.noarch.rpm: Curl error (28): Timeout was reached for https://epel.stl.us.ssimn.org/9/Everything/x86_64/Packages/p/python3-cheroot-8.6.0-4.el9.noarch.rpm [Failed to connect to epel.stl.us.ssimn.org port 443: Connection timed out] 2024-03-26T04:09:45.200 INFO:teuthology.orchestra.run.smithi016.stdout:(36/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 658 B/s | 172 kB 04:27 2024-03-26T04:09:45.200 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:09:45.200 INFO:teuthology.orchestra.run.smithi016.stdout:Total 113 kB/s | 30 MB 04:29 2024-03-26T04:09:45.221 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-26T04:09:45.267 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-26T04:09:45.267 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-26T04:09:45.695 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-26T04:09:45.696 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-26T04:09:47.081 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-26T04:09:47.143 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-26T04:09:47.185 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-26T04:09:47.230 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-26T04:09:47.303 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-26T04:09:47.345 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-26T04:09:47.378 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-26T04:09:47.475 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-26T04:09:47.560 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-26T04:09:47.617 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-26T04:09:47.689 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-26T04:09:47.750 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-26T04:09:47.787 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-26T04:09:47.846 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-26T04:09:47.925 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-26T04:09:48.137 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-26T04:09:48.184 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-26T04:09:48.232 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-26T04:09:48.273 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-26T04:09:48.303 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-26T04:09:48.382 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-26T04:09:48.422 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-26T04:09:48.455 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-26T04:09:48.496 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-26T04:09:48.540 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-26T04:09:48.582 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-26T04:09:48.675 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-26T04:09:48.714 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-26T04:09:48.788 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-26T04:09:48.929 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-26T04:09:49.052 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-26T04:09:49.221 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1775.g5e70124b.el9. 31/36 2024-03-26T04:09:49.252 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 32/36 2024-03-26T04:09:49.276 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 32/36 2024-03-26T04:09:49.276 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:09:49.276 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-26T04:09:49.276 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T04:09:49.276 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-26T04:09:49.276 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:09:50.796 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-osd-2:19.0.0-1775.g5e70124b.el9.x86_64 33/36 2024-03-26T04:09:50.822 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-osd-2:19.0.0-1775.g5e70124b.el9.x86_64 33/36 2024-03-26T04:09:50.822 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:09:50.822 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-26T04:09:50.822 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T04:09:50.822 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-26T04:09:50.822 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:09:51.319 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mon-2:19.0.0-1775.g5e70124b.el9.x86_64 34/36 2024-03-26T04:09:51.341 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mon-2:19.0.0-1775.g5e70124b.el9.x86_64 34/36 2024-03-26T04:09:51.342 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:09:51.342 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-26T04:09:51.342 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T04:09:51.342 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-26T04:09:51.342 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:09:51.571 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mds-2:19.0.0-1775.g5e70124b.el9.x86_64 35/36 2024-03-26T04:09:51.593 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mds-2:19.0.0-1775.g5e70124b.el9.x86_64 35/36 2024-03-26T04:09:51.593 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:09:51.593 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-26T04:09:51.593 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T04:09:51.594 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-26T04:09:51.594 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:09:51.663 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-2:19.0.0-1775.g5e70124b.el9.x86_64 36/36 2024-03-26T04:09:53.334 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-2:19.0.0-1775.g5e70124b.el9.x86_64 36/36 2024-03-26T04:09:53.334 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-2:19.0.0-1775.g5e70124b.el9.x86_64 1/36 2024-03-26T04:09:53.334 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mds-2:19.0.0-1775.g5e70124b.el9.x86_64 2/36 2024-03-26T04:09:53.334 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 3/36 2024-03-26T04:09:53.334 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mon-2:19.0.0-1775.g5e70124b.el9.x86_64 4/36 2024-03-26T04:09:53.334 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-osd-2:19.0.0-1775.g5e70124b.el9.x86_64 5/36 2024-03-26T04:09:53.334 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1775.g5e70124b.el9. 6/36 2024-03-26T04:09:53.335 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-26T04:09:53.335 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-26T04:09:53.335 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-26T04:09:53.335 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-26T04:09:53.335 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-26T04:09:53.335 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-26T04:09:53.335 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-26T04:09:53.335 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-26T04:09:53.335 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-26T04:09:53.335 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-26T04:09:53.335 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-26T04:09:53.335 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-26T04:09:53.335 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-26T04:09:53.336 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-26T04:09:53.336 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-26T04:09:53.336 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-26T04:09:53.336 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-26T04:09:53.336 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-26T04:09:53.336 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-26T04:09:53.336 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-26T04:09:53.336 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-26T04:09:53.336 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-26T04:09:53.336 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-26T04:09:53.336 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-26T04:09:53.336 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-26T04:09:53.336 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-26T04:09:53.336 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-26T04:09:53.337 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-26T04:09:53.337 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-26T04:09:54.381 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-26T04:09:54.381 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:09:54.381 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-26T04:09:54.381 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:09:54.381 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mds-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:09:54.381 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:09:54.381 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-modules-core-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:09:54.381 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mon-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:09:54.381 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-osd-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:09:54.382 INFO:teuthology.orchestra.run.smithi016.stdout: lua-5.4.4-4.el9.x86_64 2024-03-26T04:09:54.382 INFO:teuthology.orchestra.run.smithi016.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-26T04:09:54.382 INFO:teuthology.orchestra.run.smithi016.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-26T04:09:54.382 INFO:teuthology.orchestra.run.smithi016.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-26T04:09:54.382 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-26T04:09:54.382 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-26T04:09:54.382 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-26T04:09:54.382 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-26T04:09:54.382 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-26T04:09:54.382 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-26T04:09:54.382 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-26T04:09:54.382 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-26T04:09:54.382 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-26T04:09:54.382 INFO:teuthology.orchestra.run.smithi016.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-26T04:09:54.383 INFO:teuthology.orchestra.run.smithi016.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-26T04:09:54.383 INFO:teuthology.orchestra.run.smithi016.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-26T04:09:54.383 INFO:teuthology.orchestra.run.smithi016.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-26T04:09:54.383 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-26T04:09:54.383 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ply-3.11-14.el9.noarch 2024-03-26T04:09:54.383 INFO:teuthology.orchestra.run.smithi016.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-26T04:09:54.383 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-26T04:09:54.383 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-26T04:09:54.383 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-26T04:09:54.383 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-26T04:09:54.383 INFO:teuthology.orchestra.run.smithi016.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-26T04:09:54.383 INFO:teuthology.orchestra.run.smithi016.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-26T04:09:54.383 INFO:teuthology.orchestra.run.smithi016.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-26T04:09:54.383 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-26T04:09:54.384 INFO:teuthology.orchestra.run.smithi016.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-26T04:09:54.384 INFO:teuthology.orchestra.run.smithi016.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-26T04:09:54.384 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:09:54.384 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-26T04:09:54.731 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-base 2024-03-26T04:09:55.298 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:05:23 ago on Tue 26 Mar 2024 04:04:32 AM UTC. 2024-03-26T04:09:55.396 INFO:teuthology.orchestra.run.smithi016.stdout:Package ceph-base-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:09:55.450 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-26T04:09:55.451 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-03-26T04:09:55.451 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-26T04:09:55.508 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install cephadm 2024-03-26T04:09:56.073 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:05:24 ago on Tue 26 Mar 2024 04:04:32 AM UTC. 2024-03-26T04:09:56.227 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-26T04:09:56.228 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-26T04:09:56.228 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-03-26T04:09:56.228 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-26T04:09:56.228 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-26T04:09:56.228 INFO:teuthology.orchestra.run.smithi016.stdout: cephadm noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 748 k 2024-03-26T04:09:56.228 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:09:56.228 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-26T04:09:56.228 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-26T04:09:56.229 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-03-26T04:09:56.229 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:09:56.229 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 748 k 2024-03-26T04:09:56.229 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 754 k 2024-03-26T04:09:56.229 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-26T04:09:56.525 INFO:teuthology.orchestra.run.smithi016.stdout:cephadm-19.0.0-1775.g5e70124b.el9.noarch.rpm 2.5 MB/s | 748 kB 00:00 2024-03-26T04:09:56.526 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:09:56.526 INFO:teuthology.orchestra.run.smithi016.stdout:Total 2.5 MB/s | 748 kB 00:00 2024-03-26T04:09:56.526 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-26T04:09:56.531 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-26T04:09:56.531 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-26T04:09:56.537 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-26T04:09:56.537 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-26T04:09:56.612 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-26T04:09:57.017 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/1 2024-03-26T04:09:57.124 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/1 2024-03-26T04:09:57.522 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/1 2024-03-26T04:09:57.784 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/1 2024-03-26T04:09:57.784 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:09:57.784 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-26T04:09:57.784 INFO:teuthology.orchestra.run.smithi016.stdout: cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:09:57.784 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:09:57.784 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-26T04:09:58.004 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-immutable-object-cache 2024-03-26T04:09:58.575 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:05:26 ago on Tue 26 Mar 2024 04:04:32 AM UTC. 2024-03-26T04:09:58.731 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-26T04:09:58.732 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-26T04:09:58.732 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2024-03-26T04:09:58.732 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-26T04:09:58.732 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-26T04:09:58.733 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 145 k 2024-03-26T04:09:58.733 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:09:58.733 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-26T04:09:58.733 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-26T04:09:58.733 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-03-26T04:09:58.733 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:09:58.733 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 145 k 2024-03-26T04:09:58.733 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 436 k 2024-03-26T04:09:58.733 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-26T04:09:59.082 INFO:teuthology.orchestra.run.smithi016.stdout:ceph-immutable-object-cache-19.0.0-1775.g5e7012 417 kB/s | 145 kB 00:00 2024-03-26T04:09:59.083 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:09:59.083 INFO:teuthology.orchestra.run.smithi016.stdout:Total 415 kB/s | 145 kB 00:00 2024-03-26T04:09:59.083 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-26T04:09:59.091 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-26T04:09:59.091 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-26T04:09:59.146 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-26T04:09:59.147 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-26T04:09:59.289 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-26T04:09:59.350 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1775.g5e70124b. 1/1 2024-03-26T04:09:59.373 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1775.g5e70124b. 1/1 2024-03-26T04:09:59.374 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:09:59.374 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-26T04:09:59.374 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:00.327 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1775.g5e70124b. 1/1 2024-03-26T04:10:00.327 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:00.327 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-26T04:10:00.327 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-immutable-object-cache-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:10:00.327 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:00.327 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-26T04:10:00.552 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-mgr 2024-03-26T04:10:01.134 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:05:29 ago on Tue 26 Mar 2024 04:04:32 AM UTC. 2024-03-26T04:10:01.235 INFO:teuthology.orchestra.run.smithi016.stdout:Package ceph-mgr-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:10:01.290 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-26T04:10:01.292 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-03-26T04:10:01.292 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-26T04:10:01.350 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-mgr-dashboard 2024-03-26T04:10:01.911 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:05:29 ago on Tue 26 Mar 2024 04:04:32 AM UTC. 2024-03-26T04:10:02.068 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-26T04:10:02.069 INFO:teuthology.orchestra.run.smithi016.stdout:====================================================================================== 2024-03-26T04:10:02.069 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-03-26T04:10:02.069 INFO:teuthology.orchestra.run.smithi016.stdout:====================================================================================== 2024-03-26T04:10:02.069 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-26T04:10:02.069 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 1.8 M 2024-03-26T04:10:02.069 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-03-26T04:10:02.069 INFO:teuthology.orchestra.run.smithi016.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-26T04:10:02.070 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 31 k 2024-03-26T04:10:02.070 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 15 k 2024-03-26T04:10:02.070 INFO:teuthology.orchestra.run.smithi016.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-26T04:10:02.070 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-26T04:10:02.070 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-26T04:10:02.070 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-26T04:10:02.070 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-26T04:10:02.070 INFO:teuthology.orchestra.run.smithi016.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-26T04:10:02.070 INFO:teuthology.orchestra.run.smithi016.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-26T04:10:02.070 INFO:teuthology.orchestra.run.smithi016.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-26T04:10:02.070 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:02.070 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-26T04:10:02.070 INFO:teuthology.orchestra.run.smithi016.stdout:====================================================================================== 2024-03-26T04:10:02.071 INFO:teuthology.orchestra.run.smithi016.stdout:Install 12 Packages 2024-03-26T04:10:02.071 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:02.071 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 6.9 M 2024-03-26T04:10:02.071 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 26 M 2024-03-26T04:10:02.071 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-26T04:10:02.562 INFO:teuthology.orchestra.run.smithi016.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1775.g5e 164 kB/s | 31 kB 00:00 2024-03-26T04:10:02.653 INFO:teuthology.orchestra.run.smithi016.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1775.g5e7 54 kB/s | 15 kB 00:00 2024-03-26T04:10:02.787 INFO:teuthology.orchestra.run.smithi016.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.5 MB/s | 1.0 MB 00:00 2024-03-26T04:10:02.812 INFO:teuthology.orchestra.run.smithi016.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.5 MB/s | 238 kB 00:00 2024-03-26T04:10:03.021 INFO:teuthology.orchestra.run.smithi016.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1775.g5e70124 2.7 MB/s | 1.8 MB 00:00 2024-03-26T04:10:03.438 INFO:teuthology.orchestra.run.smithi016.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.3 MB/s | 863 kB 00:00 2024-03-26T04:10:03.496 INFO:teuthology.orchestra.run.smithi016.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 41 kB/s | 19 kB 00:00 2024-03-26T04:10:03.764 INFO:teuthology.orchestra.run.smithi016.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 542 kB/s | 144 kB 00:00 2024-03-26T04:10:03.872 INFO:teuthology.orchestra.run.smithi016.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 284 kB/s | 31 kB 00:00 2024-03-26T04:10:04.064 INFO:teuthology.orchestra.run.smithi016.stdout:(10/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 438 kB/s | 548 kB 00:01 2024-03-26T04:10:04.131 INFO:teuthology.orchestra.run.smithi016.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 730 kB/s | 188 kB 00:00 2024-03-26T04:10:06.372 INFO:teuthology.orchestra.run.smithi016.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 711 kB/s | 2.0 MB 00:02 2024-03-26T04:10:06.379 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:10:06.379 INFO:teuthology.orchestra.run.smithi016.stdout:Total 1.6 MB/s | 6.9 MB 00:04 2024-03-26T04:10:06.398 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-26T04:10:06.418 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-26T04:10:06.418 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-26T04:10:06.607 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-26T04:10:06.607 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-26T04:10:07.126 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-26T04:10:07.199 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-26T04:10:07.242 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-26T04:10:07.290 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-26T04:10:07.319 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-26T04:10:07.427 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-26T04:10:07.559 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-26T04:10:07.650 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-26T04:10:07.700 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-26T04:10:07.736 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-26T04:10:07.768 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1775.g5e70124b.el9 10/12 2024-03-26T04:10:08.022 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1775.g5e70124b.el 11/12 2024-03-26T04:10:08.067 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1775.g5e70124b.el9.noa 12/12 2024-03-26T04:10:08.876 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1775.g5e70124b.el9.noa 12/12 2024-03-26T04:10:08.876 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1775.g5e70124b.el 1/12 2024-03-26T04:10:08.876 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1775.g5e70124b.el9.noa 2/12 2024-03-26T04:10:08.876 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1775.g5e70124b.el9 3/12 2024-03-26T04:10:08.876 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-26T04:10:08.876 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-26T04:10:08.876 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-26T04:10:08.876 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-26T04:10:08.876 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-26T04:10:08.876 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-26T04:10:08.877 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-26T04:10:08.877 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-26T04:10:09.348 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-26T04:10:09.348 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:09.348 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-26T04:10:09.348 INFO:teuthology.orchestra.run.smithi016.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-26T04:10:09.348 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-grafana-dashboards-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:10:09.348 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-dashboard-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:10:09.348 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-prometheus-alerts-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:10:09.348 INFO:teuthology.orchestra.run.smithi016.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-26T04:10:09.348 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-26T04:10:09.348 INFO:teuthology.orchestra.run.smithi016.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-26T04:10:09.348 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-26T04:10:09.348 INFO:teuthology.orchestra.run.smithi016.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-26T04:10:09.349 INFO:teuthology.orchestra.run.smithi016.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-26T04:10:09.349 INFO:teuthology.orchestra.run.smithi016.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-26T04:10:09.349 INFO:teuthology.orchestra.run.smithi016.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-26T04:10:09.349 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:09.349 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-26T04:10:09.601 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-26T04:10:10.167 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:05:38 ago on Tue 26 Mar 2024 04:04:32 AM UTC. 2024-03-26T04:10:10.323 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-26T04:10:10.324 INFO:teuthology.orchestra.run.smithi016.stdout:============================================================================================ 2024-03-26T04:10:10.324 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-03-26T04:10:10.324 INFO:teuthology.orchestra.run.smithi016.stdout:============================================================================================ 2024-03-26T04:10:10.324 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-26T04:10:10.324 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 7.4 M 2024-03-26T04:10:10.324 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-03-26T04:10:10.324 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-26T04:10:10.324 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-26T04:10:10.324 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-26T04:10:10.324 INFO:teuthology.orchestra.run.smithi016.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-26T04:10:10.324 INFO:teuthology.orchestra.run.smithi016.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-26T04:10:10.324 INFO:teuthology.orchestra.run.smithi016.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-26T04:10:10.325 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-26T04:10:10.325 INFO:teuthology.orchestra.run.smithi016.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-26T04:10:10.325 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-26T04:10:10.325 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-26T04:10:10.325 INFO:teuthology.orchestra.run.smithi016.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-26T04:10:10.325 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:10.325 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-26T04:10:10.325 INFO:teuthology.orchestra.run.smithi016.stdout:============================================================================================ 2024-03-26T04:10:10.325 INFO:teuthology.orchestra.run.smithi016.stdout:Install 12 Packages 2024-03-26T04:10:10.325 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:10.326 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 37 M 2024-03-26T04:10:10.326 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 209 M 2024-03-26T04:10:10.326 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-26T04:10:10.655 INFO:teuthology.orchestra.run.smithi016.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 236 kB/s | 32 kB 00:00 2024-03-26T04:10:10.688 INFO:teuthology.orchestra.run.smithi016.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 535 kB/s | 17 kB 00:00 2024-03-26T04:10:10.772 INFO:teuthology.orchestra.run.smithi016.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-03-26T04:10:10.797 INFO:teuthology.orchestra.run.smithi016.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 319 kB/s | 34 kB 00:00 2024-03-26T04:10:10.956 INFO:teuthology.orchestra.run.smithi016.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 26 MB/s | 4.7 MB 00:00 2024-03-26T04:10:11.031 INFO:teuthology.orchestra.run.smithi016.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 21 MB/s | 4.8 MB 00:00 2024-03-26T04:10:11.065 INFO:teuthology.orchestra.run.smithi016.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.0 MB/s | 217 kB 00:00 2024-03-26T04:10:11.357 INFO:teuthology.orchestra.run.smithi016.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 48 MB/s | 16 MB 00:00 2024-03-26T04:10:11.399 INFO:teuthology.orchestra.run.smithi016.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.4 MB/s | 803 kB 00:00 2024-03-26T04:10:11.508 INFO:teuthology.orchestra.run.smithi016.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-03-26T04:10:11.624 INFO:teuthology.orchestra.run.smithi016.stdout:(11/12): ceph-mgr-diskprediction-local-19.0.0-1 6.7 MB/s | 7.4 MB 00:01 2024-03-26T04:10:11.991 INFO:teuthology.orchestra.run.smithi016.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 415 kB/s | 245 kB 00:00 2024-03-26T04:10:11.993 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:10:11.994 INFO:teuthology.orchestra.run.smithi016.stdout:Total 22 MB/s | 37 MB 00:01 2024-03-26T04:10:12.037 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-26T04:10:12.068 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-26T04:10:12.068 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-26T04:10:12.430 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-26T04:10:12.431 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-26T04:10:13.216 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-26T04:10:13.298 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-26T04:10:13.329 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-26T04:10:13.438 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-26T04:10:13.474 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-26T04:10:13.660 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-26T04:10:13.690 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-26T04:10:13.769 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-26T04:10:14.477 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-26T04:10:14.591 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-26T04:10:15.909 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-26T04:10:16.990 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-26T04:10:17.024 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1775.g5e701 12/12 2024-03-26T04:10:19.579 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1775.g5e701 12/12 2024-03-26T04:10:19.579 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1775.g5e701 1/12 2024-03-26T04:10:19.579 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-26T04:10:19.579 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-26T04:10:19.579 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-26T04:10:19.580 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-26T04:10:19.580 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-26T04:10:19.580 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-26T04:10:19.580 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-26T04:10:19.580 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-26T04:10:19.580 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-26T04:10:19.580 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-26T04:10:20.058 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-26T04:10:20.058 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:20.058 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-26T04:10:20.058 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:10:20.058 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-26T04:10:20.058 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-26T04:10:20.058 INFO:teuthology.orchestra.run.smithi016.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-26T04:10:20.059 INFO:teuthology.orchestra.run.smithi016.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-26T04:10:20.059 INFO:teuthology.orchestra.run.smithi016.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-26T04:10:20.059 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-26T04:10:20.059 INFO:teuthology.orchestra.run.smithi016.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-26T04:10:20.059 INFO:teuthology.orchestra.run.smithi016.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-26T04:10:20.059 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-26T04:10:20.059 INFO:teuthology.orchestra.run.smithi016.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-26T04:10:20.059 INFO:teuthology.orchestra.run.smithi016.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-26T04:10:20.059 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:20.059 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-26T04:10:20.444 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-mgr-rook 2024-03-26T04:10:21.024 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:05:49 ago on Tue 26 Mar 2024 04:04:32 AM UTC. 2024-03-26T04:10:21.184 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-26T04:10:21.185 INFO:teuthology.orchestra.run.smithi016.stdout:======================================================================================== 2024-03-26T04:10:21.186 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-03-26T04:10:21.186 INFO:teuthology.orchestra.run.smithi016.stdout:======================================================================================== 2024-03-26T04:10:21.186 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-26T04:10:21.186 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-rook noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 49 k 2024-03-26T04:10:21.186 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-03-26T04:10:21.186 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-26T04:10:21.186 INFO:teuthology.orchestra.run.smithi016.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-26T04:10:21.186 INFO:teuthology.orchestra.run.smithi016.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-26T04:10:21.186 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-26T04:10:21.186 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-26T04:10:21.186 INFO:teuthology.orchestra.run.smithi016.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-26T04:10:21.186 INFO:teuthology.orchestra.run.smithi016.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-26T04:10:21.186 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-26T04:10:21.187 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-26T04:10:21.187 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-26T04:10:21.187 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-26T04:10:21.187 INFO:teuthology.orchestra.run.smithi016.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-26T04:10:21.187 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:21.187 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-26T04:10:21.187 INFO:teuthology.orchestra.run.smithi016.stdout:======================================================================================== 2024-03-26T04:10:21.187 INFO:teuthology.orchestra.run.smithi016.stdout:Install 13 Packages 2024-03-26T04:10:21.187 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:21.188 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 2.0 M 2024-03-26T04:10:21.188 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 26 M 2024-03-26T04:10:21.188 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-26T04:10:21.516 INFO:teuthology.orchestra.run.smithi016.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 138 kB/s | 18 kB 00:00 2024-03-26T04:10:21.540 INFO:teuthology.orchestra.run.smithi016.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 156 kB/s | 24 kB 00:00 2024-03-26T04:10:21.582 INFO:teuthology.orchestra.run.smithi016.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-26T04:10:21.608 INFO:teuthology.orchestra.run.smithi016.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-26T04:10:21.633 INFO:teuthology.orchestra.run.smithi016.stdout:(5/13): ceph-mgr-rook-19.0.0-1775.g5e70124b.el9 198 kB/s | 49 kB 00:00 2024-03-26T04:10:21.667 INFO:teuthology.orchestra.run.smithi016.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.5 MB/s | 211 kB 00:00 2024-03-26T04:10:21.692 INFO:teuthology.orchestra.run.smithi016.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 512 kB/s | 43 kB 00:00 2024-03-26T04:10:21.959 INFO:teuthology.orchestra.run.smithi016.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 51 kB/s | 15 kB 00:00 2024-03-26T04:10:21.984 INFO:teuthology.orchestra.run.smithi016.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 101 kB/s | 32 kB 00:00 2024-03-26T04:10:22.226 INFO:teuthology.orchestra.run.smithi016.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 379 kB/s | 203 kB 00:00 2024-03-26T04:10:22.343 INFO:teuthology.orchestra.run.smithi016.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 165 kB/s | 59 kB 00:00 2024-03-26T04:10:22.611 INFO:teuthology.orchestra.run.smithi016.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 234 kB/s | 90 kB 00:00 2024-03-26T04:10:22.727 INFO:teuthology.orchestra.run.smithi016.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.3 MB/s | 1.0 MB 00:00 2024-03-26T04:10:22.730 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:10:22.730 INFO:teuthology.orchestra.run.smithi016.stdout:Total 1.3 MB/s | 2.0 MB 00:01 2024-03-26T04:10:22.731 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-26T04:10:22.747 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-26T04:10:22.747 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-26T04:10:22.847 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-26T04:10:22.849 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-26T04:10:23.606 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-26T04:10:23.774 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-26T04:10:23.830 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-26T04:10:23.886 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-26T04:10:23.930 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-26T04:10:23.981 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-26T04:10:24.077 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-26T04:10:24.186 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-26T04:10:24.241 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-26T04:10:25.003 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-26T04:10:25.054 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-26T04:10:25.087 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-26T04:10:25.136 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-26T04:10:25.165 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-rook-2:19.0.0-1775.g5e70124b.el9.noarch 13/13 2024-03-26T04:10:25.922 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1775.g5e70124b.el9.noarch 13/13 2024-03-26T04:10:25.923 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1775.g5e70124b.el9.noarch 1/13 2024-03-26T04:10:25.923 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-26T04:10:25.923 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-26T04:10:25.923 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-26T04:10:25.923 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-26T04:10:25.923 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-26T04:10:25.923 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-26T04:10:25.923 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-26T04:10:25.923 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-26T04:10:25.923 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-26T04:10:25.923 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-26T04:10:25.923 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-26T04:10:26.305 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-26T04:10:26.305 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:26.305 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-26T04:10:26.305 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-rook-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:10:26.305 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-26T04:10:26.305 INFO:teuthology.orchestra.run.smithi016.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-26T04:10:26.305 INFO:teuthology.orchestra.run.smithi016.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-26T04:10:26.305 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-26T04:10:26.305 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-26T04:10:26.305 INFO:teuthology.orchestra.run.smithi016.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-26T04:10:26.305 INFO:teuthology.orchestra.run.smithi016.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-26T04:10:26.305 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-26T04:10:26.306 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-26T04:10:26.306 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-26T04:10:26.306 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-26T04:10:26.306 INFO:teuthology.orchestra.run.smithi016.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-26T04:10:26.306 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:26.306 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-26T04:10:26.559 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-mgr-cephadm 2024-03-26T04:10:27.127 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:05:55 ago on Tue 26 Mar 2024 04:04:32 AM UTC. 2024-03-26T04:10:27.282 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-26T04:10:27.283 INFO:teuthology.orchestra.run.smithi016.stdout:======================================================================================== 2024-03-26T04:10:27.283 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-03-26T04:10:27.283 INFO:teuthology.orchestra.run.smithi016.stdout:======================================================================================== 2024-03-26T04:10:27.283 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-26T04:10:27.283 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 138 k 2024-03-26T04:10:27.283 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-03-26T04:10:27.284 INFO:teuthology.orchestra.run.smithi016.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-26T04:10:27.284 INFO:teuthology.orchestra.run.smithi016.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-26T04:10:27.284 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-26T04:10:27.284 INFO:teuthology.orchestra.run.smithi016.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-26T04:10:27.284 INFO:teuthology.orchestra.run.smithi016.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-26T04:10:27.284 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:27.284 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-26T04:10:27.284 INFO:teuthology.orchestra.run.smithi016.stdout:======================================================================================== 2024-03-26T04:10:27.284 INFO:teuthology.orchestra.run.smithi016.stdout:Install 6 Packages 2024-03-26T04:10:27.284 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:27.285 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 6.8 M 2024-03-26T04:10:27.285 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 33 M 2024-03-26T04:10:27.285 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-26T04:10:27.863 INFO:teuthology.orchestra.run.smithi016.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-26T04:10:27.888 INFO:teuthology.orchestra.run.smithi016.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1775.g5e70124b.e 483 kB/s | 138 kB 00:00 2024-03-26T04:10:27.922 INFO:teuthology.orchestra.run.smithi016.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 776 kB/s | 248 kB 00:00 2024-03-26T04:10:28.148 INFO:teuthology.orchestra.run.smithi016.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 223 kB/s | 58 kB 00:00 2024-03-26T04:10:28.173 INFO:teuthology.orchestra.run.smithi016.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 204 kB/s | 51 kB 00:00 2024-03-26T04:10:28.365 INFO:teuthology.orchestra.run.smithi016.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.1 MB/s | 548 kB 00:00 2024-03-26T04:10:28.366 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:10:28.366 INFO:teuthology.orchestra.run.smithi016.stdout:Total 6.3 MB/s | 6.8 MB 00:01 2024-03-26T04:10:28.381 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-26T04:10:28.393 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-26T04:10:28.393 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-26T04:10:28.488 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-26T04:10:28.489 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-26T04:10:28.788 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-26T04:10:28.974 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-26T04:10:29.034 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-26T04:10:29.585 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-26T04:10:29.671 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-26T04:10:29.732 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-26T04:10:29.762 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 6/6 2024-03-26T04:10:30.540 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 6/6 2024-03-26T04:10:30.541 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 1/6 2024-03-26T04:10:30.541 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-26T04:10:30.541 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-26T04:10:30.541 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-26T04:10:30.541 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-26T04:10:30.846 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-26T04:10:30.846 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:30.846 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-26T04:10:30.846 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-cephadm-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:10:30.846 INFO:teuthology.orchestra.run.smithi016.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-26T04:10:30.847 INFO:teuthology.orchestra.run.smithi016.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-26T04:10:30.847 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-26T04:10:30.847 INFO:teuthology.orchestra.run.smithi016.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-26T04:10:30.847 INFO:teuthology.orchestra.run.smithi016.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-26T04:10:30.847 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:30.847 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-26T04:10:31.062 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-fuse 2024-03-26T04:10:31.634 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:05:59 ago on Tue 26 Mar 2024 04:04:32 AM UTC. 2024-03-26T04:10:31.791 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-26T04:10:31.792 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-26T04:10:31.792 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-03-26T04:10:31.792 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-26T04:10:31.792 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-26T04:10:31.792 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-fuse x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 888 k 2024-03-26T04:10:31.793 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-03-26T04:10:31.793 INFO:teuthology.orchestra.run.smithi016.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-26T04:10:31.793 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:31.793 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-26T04:10:31.793 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-26T04:10:31.793 INFO:teuthology.orchestra.run.smithi016.stdout:Install 2 Packages 2024-03-26T04:10:31.793 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:31.793 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 967 k 2024-03-26T04:10:31.793 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 2.8 M 2024-03-26T04:10:31.793 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-26T04:10:31.925 INFO:teuthology.orchestra.run.smithi016.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 597 kB/s | 79 kB 00:00 2024-03-26T04:10:32.242 INFO:teuthology.orchestra.run.smithi016.stdout:(2/2): ceph-fuse-19.0.0-1775.g5e70124b.el9.x86_ 1.9 MB/s | 888 kB 00:00 2024-03-26T04:10:32.244 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:10:32.244 INFO:teuthology.orchestra.run.smithi016.stdout:Total 2.1 MB/s | 967 kB 00:00 2024-03-26T04:10:32.244 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-26T04:10:32.266 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-26T04:10:32.266 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-26T04:10:32.342 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-26T04:10:32.342 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-26T04:10:32.526 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-26T04:10:32.688 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-26T04:10:32.770 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 2/2 2024-03-26T04:10:33.466 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 2/2 2024-03-26T04:10:33.467 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 1/2 2024-03-26T04:10:33.785 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-26T04:10:33.785 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:33.785 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-26T04:10:33.785 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-26T04:10:33.786 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:33.786 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-26T04:10:34.074 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-volume 2024-03-26T04:10:34.652 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:06:02 ago on Tue 26 Mar 2024 04:04:32 AM UTC. 2024-03-26T04:10:34.808 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-26T04:10:34.809 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-26T04:10:34.809 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2024-03-26T04:10:34.809 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-26T04:10:34.809 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-26T04:10:34.809 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-volume noarch 2:19.0.0-1775.g5e70124b.el9 ceph-noarch 264 k 2024-03-26T04:10:34.809 INFO:teuthology.orchestra.run.smithi016.stdout:Installing dependencies: 2024-03-26T04:10:34.809 INFO:teuthology.orchestra.run.smithi016.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-26T04:10:34.809 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:34.809 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-26T04:10:34.809 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-26T04:10:34.810 INFO:teuthology.orchestra.run.smithi016.stdout:Install 2 Packages 2024-03-26T04:10:34.810 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:34.810 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 332 k 2024-03-26T04:10:34.810 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 1.5 M 2024-03-26T04:10:34.810 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-26T04:10:34.940 INFO:teuthology.orchestra.run.smithi016.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 531 kB/s | 69 kB 00:00 2024-03-26T04:10:35.090 INFO:teuthology.orchestra.run.smithi016.stdout:(2/2): ceph-volume-19.0.0-1775.g5e70124b.el9.no 944 kB/s | 264 kB 00:00 2024-03-26T04:10:35.091 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:10:35.091 INFO:teuthology.orchestra.run.smithi016.stdout:Total 1.2 MB/s | 332 kB 00:00 2024-03-26T04:10:35.091 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-26T04:10:35.101 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-26T04:10:35.101 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-26T04:10:35.145 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-26T04:10:35.145 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-26T04:10:35.294 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-26T04:10:35.471 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-26T04:10:35.517 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : ceph-volume-2:19.0.0-1775.g5e70124b.el9.noarch 2/2 2024-03-26T04:10:35.533 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: ceph-volume-2:19.0.0-1775.g5e70124b.el9.noarch 2/2 2024-03-26T04:10:35.533 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:10:35.533 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-26T04:10:35.533 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:36.242 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : ceph-volume-2:19.0.0-1775.g5e70124b.el9.noarch 1/2 2024-03-26T04:10:36.581 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-26T04:10:36.581 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:36.581 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-26T04:10:36.581 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-volume-2:19.0.0-1775.g5e70124b.el9.noarch 2024-03-26T04:10:36.581 INFO:teuthology.orchestra.run.smithi016.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-26T04:10:36.581 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:36.581 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-26T04:10:36.926 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install librados-devel 2024-03-26T04:10:37.501 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:06:05 ago on Tue 26 Mar 2024 04:04:32 AM UTC. 2024-03-26T04:10:37.656 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-26T04:10:37.657 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-26T04:10:37.657 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2024-03-26T04:10:37.657 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-26T04:10:37.658 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-26T04:10:37.658 INFO:teuthology.orchestra.run.smithi016.stdout: librados-devel x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 127 k 2024-03-26T04:10:37.658 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:37.658 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-26T04:10:37.658 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-26T04:10:37.658 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-03-26T04:10:37.658 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:37.658 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 127 k 2024-03-26T04:10:37.658 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 456 k 2024-03-26T04:10:37.658 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-26T04:10:37.882 INFO:teuthology.orchestra.run.smithi016.stdout:librados-devel-19.0.0-1775.g5e70124b.el9.x86_64 569 kB/s | 127 kB 00:00 2024-03-26T04:10:37.882 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:10:37.883 INFO:teuthology.orchestra.run.smithi016.stdout:Total 565 kB/s | 127 kB 00:00 2024-03-26T04:10:37.883 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-26T04:10:37.888 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-26T04:10:37.888 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-26T04:10:37.946 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-26T04:10:37.946 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-26T04:10:38.109 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-26T04:10:38.208 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librados-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:10:38.651 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: librados-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:10:38.943 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librados-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:10:38.944 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:38.944 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-26T04:10:38.944 INFO:teuthology.orchestra.run.smithi016.stdout: librados-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:10:38.944 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:38.944 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-26T04:10:39.164 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install libcephfs2 2024-03-26T04:10:39.742 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:06:07 ago on Tue 26 Mar 2024 04:04:32 AM UTC. 2024-03-26T04:10:39.843 INFO:teuthology.orchestra.run.smithi016.stdout:Package libcephfs2-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:10:39.900 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-26T04:10:39.901 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-03-26T04:10:39.901 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-26T04:10:39.954 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install libcephfs-devel 2024-03-26T04:10:40.523 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:06:08 ago on Tue 26 Mar 2024 04:04:32 AM UTC. 2024-03-26T04:10:40.679 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-26T04:10:40.680 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-26T04:10:40.680 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2024-03-26T04:10:40.680 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-26T04:10:40.680 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-26T04:10:40.680 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs-devel x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 31 k 2024-03-26T04:10:40.680 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:40.680 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-26T04:10:40.680 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-26T04:10:40.680 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-03-26T04:10:40.680 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:40.681 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 31 k 2024-03-26T04:10:40.681 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 140 k 2024-03-26T04:10:40.681 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-26T04:10:40.927 INFO:teuthology.orchestra.run.smithi016.stdout:libcephfs-devel-19.0.0-1775.g5e70124b.el9.x86_6 127 kB/s | 31 kB 00:00 2024-03-26T04:10:40.927 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:10:40.927 INFO:teuthology.orchestra.run.smithi016.stdout:Total 126 kB/s | 31 kB 00:00 2024-03-26T04:10:40.928 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-26T04:10:40.931 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-26T04:10:40.931 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-26T04:10:40.962 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-26T04:10:40.962 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-26T04:10:41.074 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-26T04:10:41.203 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcephfs-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:10:41.644 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:10:41.917 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcephfs-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:10:41.917 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:41.917 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-26T04:10:41.917 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs-devel-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:10:41.917 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:41.918 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-26T04:10:42.096 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install librados2 2024-03-26T04:10:42.667 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:06:10 ago on Tue 26 Mar 2024 04:04:32 AM UTC. 2024-03-26T04:10:42.767 INFO:teuthology.orchestra.run.smithi016.stdout:Package librados2-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:10:42.821 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-26T04:10:42.822 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-03-26T04:10:42.822 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-26T04:10:42.870 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install librbd1 2024-03-26T04:10:43.443 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:06:11 ago on Tue 26 Mar 2024 04:04:32 AM UTC. 2024-03-26T04:10:43.549 INFO:teuthology.orchestra.run.smithi016.stdout:Package librbd1-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:10:43.605 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-26T04:10:43.606 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-03-26T04:10:43.606 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-26T04:10:43.650 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install python3-rados 2024-03-26T04:10:44.221 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:06:12 ago on Tue 26 Mar 2024 04:04:32 AM UTC. 2024-03-26T04:10:44.319 INFO:teuthology.orchestra.run.smithi016.stdout:Package python3-rados-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:10:44.374 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-26T04:10:44.375 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-03-26T04:10:44.375 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-26T04:10:44.429 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install python3-rgw 2024-03-26T04:10:45.002 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:06:13 ago on Tue 26 Mar 2024 04:04:32 AM UTC. 2024-03-26T04:10:45.101 INFO:teuthology.orchestra.run.smithi016.stdout:Package python3-rgw-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:10:45.156 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-26T04:10:45.157 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-03-26T04:10:45.157 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-26T04:10:45.215 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install python3-cephfs 2024-03-26T04:10:45.784 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:06:13 ago on Tue 26 Mar 2024 04:04:32 AM UTC. 2024-03-26T04:10:45.886 INFO:teuthology.orchestra.run.smithi016.stdout:Package python3-cephfs-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:10:45.941 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-26T04:10:45.942 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-03-26T04:10:45.942 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-26T04:10:46.007 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install python3-rbd 2024-03-26T04:10:46.582 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:06:14 ago on Tue 26 Mar 2024 04:04:32 AM UTC. 2024-03-26T04:10:46.682 INFO:teuthology.orchestra.run.smithi016.stdout:Package python3-rbd-2:19.0.0-1775.g5e70124b.el9.x86_64 is already installed. 2024-03-26T04:10:46.737 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-26T04:10:46.738 INFO:teuthology.orchestra.run.smithi016.stdout:Nothing to do. 2024-03-26T04:10:46.739 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-26T04:10:46.784 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install rbd-fuse 2024-03-26T04:10:47.355 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:06:15 ago on Tue 26 Mar 2024 04:04:32 AM UTC. 2024-03-26T04:10:47.510 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-26T04:10:47.513 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-26T04:10:47.514 INFO:teuthology.orchestra.run.smithi016.stdout: Package Architecture Version Repository Size 2024-03-26T04:10:47.514 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-26T04:10:47.514 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-26T04:10:47.514 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-fuse x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 85 k 2024-03-26T04:10:47.514 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:47.515 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-26T04:10:47.515 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-26T04:10:47.515 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-03-26T04:10:47.515 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:47.515 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 85 k 2024-03-26T04:10:47.515 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 231 k 2024-03-26T04:10:47.515 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-26T04:10:47.822 INFO:teuthology.orchestra.run.smithi016.stdout:rbd-fuse-19.0.0-1775.g5e70124b.el9.x86_64.rpm 274 kB/s | 85 kB 00:00 2024-03-26T04:10:47.822 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:10:47.823 INFO:teuthology.orchestra.run.smithi016.stdout:Total 273 kB/s | 85 kB 00:00 2024-03-26T04:10:47.823 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-26T04:10:47.830 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-26T04:10:47.830 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-26T04:10:47.888 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-26T04:10:47.889 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-26T04:10:48.033 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-26T04:10:48.148 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:10:48.546 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:10:48.834 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:10:48.834 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:48.834 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-26T04:10:48.834 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-fuse-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:10:48.834 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:48.834 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-26T04:10:49.040 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install rbd-mirror 2024-03-26T04:10:49.617 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:06:17 ago on Tue 26 Mar 2024 04:04:32 AM UTC. 2024-03-26T04:10:49.773 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-26T04:10:49.774 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-26T04:10:49.774 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repo Size 2024-03-26T04:10:49.774 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-26T04:10:49.774 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-26T04:10:49.774 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-mirror x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 3.1 M 2024-03-26T04:10:49.774 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:49.774 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-26T04:10:49.774 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-26T04:10:49.774 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-03-26T04:10:49.774 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:49.775 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 3.1 M 2024-03-26T04:10:49.775 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 13 M 2024-03-26T04:10:49.775 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-26T04:10:50.209 INFO:teuthology.orchestra.run.smithi016.stdout:rbd-mirror-19.0.0-1775.g5e70124b.el9.x86_64.rpm 7.1 MB/s | 3.1 MB 00:00 2024-03-26T04:10:50.209 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:10:50.210 INFO:teuthology.orchestra.run.smithi016.stdout:Total 7.1 MB/s | 3.1 MB 00:00 2024-03-26T04:10:50.210 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-26T04:10:50.220 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-26T04:10:50.220 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-26T04:10:50.295 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-26T04:10:50.296 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-26T04:10:50.737 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-26T04:10:50.793 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-mirror-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:10:50.817 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:10:50.817 INFO:teuthology.orchestra.run.smithi016.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-26T04:10:50.817 INFO:teuthology.orchestra.run.smithi016.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-26T04:10:50.817 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T04:10:50.817 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-26T04:10:50.817 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:52.078 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-mirror-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:10:52.078 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:52.078 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-26T04:10:52.078 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-mirror-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:10:52.078 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:52.078 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-26T04:10:52.297 DEBUG:teuthology.orchestra.run.smithi016:> sudo yum -y install rbd-nbd 2024-03-26T04:10:52.880 INFO:teuthology.orchestra.run.smithi016.stdout:Last metadata expiration check: 0:06:20 ago on Tue 26 Mar 2024 04:04:32 AM UTC. 2024-03-26T04:10:53.034 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies resolved. 2024-03-26T04:10:53.035 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-26T04:10:53.035 INFO:teuthology.orchestra.run.smithi016.stdout: Package Architecture Version Repository Size 2024-03-26T04:10:53.035 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-26T04:10:53.035 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2024-03-26T04:10:53.035 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-nbd x86_64 2:19.0.0-1775.g5e70124b.el9 ceph 171 k 2024-03-26T04:10:53.035 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:53.035 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2024-03-26T04:10:53.035 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2024-03-26T04:10:53.035 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2024-03-26T04:10:53.035 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:53.036 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 171 k 2024-03-26T04:10:53.036 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 494 k 2024-03-26T04:10:53.036 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading Packages: 2024-03-26T04:10:53.263 INFO:teuthology.orchestra.run.smithi016.stdout:rbd-nbd-19.0.0-1775.g5e70124b.el9.x86_64.rpm 751 kB/s | 171 kB 00:00 2024-03-26T04:10:53.264 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:10:53.264 INFO:teuthology.orchestra.run.smithi016.stdout:Total 746 kB/s | 171 kB 00:00 2024-03-26T04:10:53.264 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2024-03-26T04:10:53.273 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction check succeeded. 2024-03-26T04:10:53.273 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2024-03-26T04:10:53.333 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded. 2024-03-26T04:10:53.333 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2024-03-26T04:10:53.490 INFO:teuthology.orchestra.run.smithi016.stdout: Preparing : 1/1 2024-03-26T04:10:53.599 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : rbd-nbd-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:10:54.036 INFO:teuthology.orchestra.run.smithi016.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:10:54.335 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : rbd-nbd-2:19.0.0-1775.g5e70124b.el9.x86_64 1/1 2024-03-26T04:10:54.335 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:54.335 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2024-03-26T04:10:54.335 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-nbd-2:19.0.0-1775.g5e70124b.el9.x86_64 2024-03-26T04:10:54.335 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:10:54.335 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2024-03-26T04:10:54.555 DEBUG:teuthology.parallel:result is None 2024-03-26T04:10:54.555 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:10:54.702 DEBUG:teuthology.orchestra.run.smithi016:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-26T04:10:54.722 INFO:teuthology.orchestra.run.smithi016.stdout:19.0.0-1775.g5e70124b.el9 2024-03-26T04:10:54.723 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1775.g5e70124b.el9 2024-03-26T04:10:54.723 INFO:teuthology.task.install:The correct ceph version 19.0.0-1775.g5e70124b is installed. 2024-03-26T04:10:54.725 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:10:54.950 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-26T04:10:54.977 INFO:teuthology.orchestra.run.smithi183.stdout:19.0.0-1775.g5e70124b.el9 2024-03-26T04:10:54.977 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1775.g5e70124b.el9 2024-03-26T04:10:54.977 INFO:teuthology.task.install:The correct ceph version 19.0.0-1775.g5e70124b is installed. 2024-03-26T04:10:54.979 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-26T04:10:54.979 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-26T04:10:54.980 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-26T04:10:55.009 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-26T04:10:55.010 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-26T04:10:55.050 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-26T04:10:55.051 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-26T04:10:55.051 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/daemon-helper 2024-03-26T04:10:55.079 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-26T04:10:55.147 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-26T04:10:55.147 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/daemon-helper 2024-03-26T04:10:55.179 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-26T04:10:55.249 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-26T04:10:55.250 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-26T04:10:55.250 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-26T04:10:55.280 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-26T04:10:55.347 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-26T04:10:55.347 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-26T04:10:55.377 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-26T04:10:55.448 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-26T04:10:55.449 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-26T04:10:55.449 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/stdin-killer 2024-03-26T04:10:55.476 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-26T04:10:55.544 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-26T04:10:55.544 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/stdin-killer 2024-03-26T04:10:55.575 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-26T04:10:55.644 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-26T04:10:55.744 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client': {'debug ms': 1}, 'global': {'mon election default strategy': 1, 'ms bind msgr2': False, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon warn on pool no app': False}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd class default list': '*', 'osd class load list': '*', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'reached quota', 'but it is still running', 'overall HEALTH_', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(PG_DEGRADED\\)'], 'mon_bind_msgr2': False, 'sha1': '5e70124b46c2a671efcaae4690cd9ea72670710a', 'cephadm_mode': 'cephadm-package'} 2024-03-26T04:10:55.744 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:10:55.744 INFO:tasks.cephadm:Cluster fsid is d7d59826-eb26-11ee-95d0-87774f69a715 2024-03-26T04:10:55.745 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-26T04:10:55.745 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '[v1:172.21.15.16:6789]', 'mon.c': '[v1:172.21.15.16:6790]', 'mon.b': '[v1:172.21.15.183:6789]'} 2024-03-26T04:10:55.745 INFO:tasks.cephadm:First mon is mon.a on smithi016 2024-03-26T04:10:55.745 INFO:tasks.cephadm:First mgr is y 2024-03-26T04:10:55.745 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-26T04:10:55.745 DEBUG:teuthology.orchestra.run.smithi016:> sudo hostname $(hostname -s) 2024-03-26T04:10:55.774 DEBUG:teuthology.orchestra.run.smithi183:> sudo hostname $(hostname -s) 2024-03-26T04:10:55.813 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a on all hosts... 2024-03-26T04:10:55.813 DEBUG:teuthology.orchestra.run.smithi016:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a pull 2024-03-26T04:10:55.816 DEBUG:teuthology.orchestra.run.smithi183:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a pull 2024-03-26T04:10:56.572 INFO:teuthology.orchestra.run.smithi183.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a... 2024-03-26T04:10:56.676 INFO:teuthology.orchestra.run.smithi016.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a... 2024-03-26T04:11:29.521 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-03-26T04:11:29.521 INFO:teuthology.orchestra.run.smithi183.stdout: "ceph_version": "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)", 2024-03-26T04:11:29.521 INFO:teuthology.orchestra.run.smithi183.stdout: "image_id": "5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767", 2024-03-26T04:11:29.521 INFO:teuthology.orchestra.run.smithi183.stdout: "repo_digests": [ 2024-03-26T04:11:29.522 INFO:teuthology.orchestra.run.smithi183.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2" 2024-03-26T04:11:29.522 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-03-26T04:11:29.522 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-03-26T04:11:30.657 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-03-26T04:11:30.657 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev)", 2024-03-26T04:11:30.657 INFO:teuthology.orchestra.run.smithi016.stdout: "image_id": "5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767", 2024-03-26T04:11:30.657 INFO:teuthology.orchestra.run.smithi016.stdout: "repo_digests": [ 2024-03-26T04:11:30.657 INFO:teuthology.orchestra.run.smithi016.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2" 2024-03-26T04:11:30.658 INFO:teuthology.orchestra.run.smithi016.stdout: ] 2024-03-26T04:11:30.658 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-03-26T04:11:30.683 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /etc/ceph 2024-03-26T04:11:30.739 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /etc/ceph 2024-03-26T04:11:30.774 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 777 /etc/ceph 2024-03-26T04:11:30.800 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 777 /etc/ceph 2024-03-26T04:11:30.844 INFO:tasks.cephadm:Writing seed config... 2024-03-26T04:11:30.845 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-26T04:11:30.845 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-26T04:11:30.845 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-26T04:11:30.845 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-03-26T04:11:30.845 INFO:tasks.cephadm: override: [global] ms bind msgr2 = False 2024-03-26T04:11:30.845 INFO:tasks.cephadm: override: [global] ms type = async 2024-03-26T04:11:30.846 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-26T04:11:30.846 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-26T04:11:30.846 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-26T04:11:30.846 INFO:tasks.cephadm: override: [mon] mon warn on pool no app = False 2024-03-26T04:11:30.846 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-26T04:11:30.846 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-26T04:11:30.846 INFO:tasks.cephadm: override: [osd] osd class default list = * 2024-03-26T04:11:30.846 INFO:tasks.cephadm: override: [osd] osd class load list = * 2024-03-26T04:11:30.846 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-26T04:11:30.847 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-26T04:11:30.847 DEBUG:teuthology.orchestra.run.smithi016:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-26T04:11:30.861 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = d7d59826-eb26-11ee-95d0-87774f69a715 mon election default strategy = 1 ms bind msgr2 = False ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd class default list = * osd class load list = * [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon warn on pool no app = False [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] debug ms = 1 2024-03-26T04:11:30.862 DEBUG:teuthology.orchestra.run.smithi016:mon.a> sudo journalctl -f -n 0 -u ceph-d7d59826-eb26-11ee-95d0-87774f69a715@mon.a.service 2024-03-26T04:11:30.904 DEBUG:teuthology.orchestra.run.smithi016:mgr.y> sudo journalctl -f -n 0 -u ceph-d7d59826-eb26-11ee-95d0-87774f69a715@mgr.y.service 2024-03-26T04:11:30.947 INFO:tasks.cephadm:Bootstrapping... 2024-03-26T04:11:30.947 DEBUG:teuthology.orchestra.run.smithi016:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a -v bootstrap --fsid d7d59826-eb26-11ee-95d0-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-addrv '[v1:172.21.15.16:6789]' --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-26T04:11:31.142 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2024-03-26T04:11:31.142 INFO:teuthology.orchestra.run.smithi016.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a', '-v', 'bootstrap', '--fsid', 'd7d59826-eb26-11ee-95d0-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-addrv', '[v1:172.21.15.16:6789]', '--skip-admin-label'] 2024-03-26T04:11:31.166 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-26T04:11:31.166 INFO:teuthology.orchestra.run.smithi016.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-26T04:11:31.167 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying podman|docker is present... 2024-03-26T04:11:31.191 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-26T04:11:31.191 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying lvm2 is present... 2024-03-26T04:11:31.191 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying time synchronization is in place... 2024-03-26T04:11:31.198 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-26T04:11:31.198 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-26T04:11:31.204 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-26T04:11:31.204 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-03-26T04:11:31.210 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout enabled 2024-03-26T04:11:31.216 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout active 2024-03-26T04:11:31.216 INFO:teuthology.orchestra.run.smithi016.stdout:Unit chronyd.service is enabled and running 2024-03-26T04:11:31.216 INFO:teuthology.orchestra.run.smithi016.stdout:Repeating the final host check... 2024-03-26T04:11:31.239 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stdout 4.9.4-dev 2024-03-26T04:11:31.239 INFO:teuthology.orchestra.run.smithi016.stdout:podman (/bin/podman) version 4.9.4 is present 2024-03-26T04:11:31.239 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl is present 2024-03-26T04:11:31.239 INFO:teuthology.orchestra.run.smithi016.stdout:lvcreate is present 2024-03-26T04:11:31.245 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-26T04:11:31.245 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-26T04:11:31.251 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-26T04:11:31.251 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-03-26T04:11:31.257 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout enabled 2024-03-26T04:11:31.263 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout active 2024-03-26T04:11:31.264 INFO:teuthology.orchestra.run.smithi016.stdout:Unit chronyd.service is enabled and running 2024-03-26T04:11:31.264 INFO:teuthology.orchestra.run.smithi016.stdout:Host looks OK 2024-03-26T04:11:31.264 INFO:teuthology.orchestra.run.smithi016.stdout:Cluster fsid: d7d59826-eb26-11ee-95d0-87774f69a715 2024-03-26T04:11:31.264 INFO:teuthology.orchestra.run.smithi016.stdout:Acquiring lock 139656890709616 on /run/cephadm/d7d59826-eb26-11ee-95d0-87774f69a715.lock 2024-03-26T04:11:31.264 INFO:teuthology.orchestra.run.smithi016.stdout:Lock 139656890709616 acquired on /run/cephadm/d7d59826-eb26-11ee-95d0-87774f69a715.lock 2024-03-26T04:11:31.264 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying IP 172.21.15.16 port 6789 ... 2024-03-26T04:11:31.265 INFO:teuthology.orchestra.run.smithi016.stdout:Base mon IP(s) is [172.21.15.16:6789], mon addrv is [v1:172.21.15.16:6789] 2024-03-26T04:11:31.267 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.16 metric 100 2024-03-26T04:11:31.267 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.16 metric 100 2024-03-26T04:11:31.270 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-26T04:11:31.270 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-26T04:11:31.270 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-26T04:11:31.272 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-26T04:11:31.272 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-26T04:11:31.272 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-26T04:11:31.272 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-26T04:11:31.272 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:1564/64 scope link noprefixroute 2024-03-26T04:11:31.272 INFO:teuthology.orchestra.run.smithi016.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-26T04:11:31.273 INFO:teuthology.orchestra.run.smithi016.stdout:Mon IP `172.21.15.16` is in CIDR network `172.21.0.0/20` 2024-03-26T04:11:31.273 INFO:teuthology.orchestra.run.smithi016.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20'] 2024-03-26T04:11:31.273 INFO:teuthology.orchestra.run.smithi016.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-26T04:11:31.274 INFO:teuthology.orchestra.run.smithi016.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a... 2024-03-26T04:11:31.711 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stdout 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 2024-03-26T04:11:31.711 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a... 2024-03-26T04:11:31.711 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Getting image source signatures 2024-03-26T04:11:31.711 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Copying blob sha256:45ddbf13772d0d6535ecfdcaa58ecfc496e5849512dfbf602c5218332a216bcc 2024-03-26T04:11:31.711 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Copying blob sha256:9516588f908cdb14b5e4062a6063db2d9291b355c3f32e2a99864608a4a7b649 2024-03-26T04:11:31.712 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Copying config sha256:5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 2024-03-26T04:11:31.712 INFO:teuthology.orchestra.run.smithi016.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-26T04:11:32.590 INFO:teuthology.orchestra.run.smithi016.stdout:ceph: stdout ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev) 2024-03-26T04:11:32.590 INFO:teuthology.orchestra.run.smithi016.stdout:Ceph version: ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev) 2024-03-26T04:11:32.590 INFO:teuthology.orchestra.run.smithi016.stdout:Extracting ceph user uid/gid from container image... 2024-03-26T04:11:33.579 INFO:teuthology.orchestra.run.smithi016.stdout:stat: stdout 167 167 2024-03-26T04:11:33.579 INFO:teuthology.orchestra.run.smithi016.stdout:Creating initial keys... 2024-03-26T04:11:34.468 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph-authtool: stdout AQD1SgJm3QT9NBAAzc5mYzJEXWXTIc52DSl6ww== 2024-03-26T04:11:35.358 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph-authtool: stdout AQD2SgJmx/DiLhAAyZzcuT1gVcH04Ew1l4dvhw== 2024-03-26T04:11:42.066 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph-authtool: stdout AQD3SgJmaV7hKhAAUyrRU2CVWy39rgtX3BflBA== 2024-03-26T04:11:42.067 INFO:teuthology.orchestra.run.smithi016.stdout:Creating initial monmap... 2024-03-26T04:11:46.701 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-26T04:11:46.701 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-26T04:11:46.701 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to d7d59826-eb26-11ee-95d0-87774f69a715 2024-03-26T04:11:46.701 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-26T04:11:46.701 INFO:teuthology.orchestra.run.smithi016.stdout:monmaptool for a [v1:172.21.15.16:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-26T04:11:46.701 INFO:teuthology.orchestra.run.smithi016.stdout:setting min_mon_release = quincy 2024-03-26T04:11:46.701 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: set fsid to d7d59826-eb26-11ee-95d0-87774f69a715 2024-03-26T04:11:46.702 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-26T04:11:46.702 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:11:46.702 INFO:teuthology.orchestra.run.smithi016.stdout:Creating mon... 2024-03-26T04:11:47.757 INFO:teuthology.orchestra.run.smithi016.stdout:create mon.a on 2024-03-26T04:11:47.982 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-26T04:11:48.126 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-26T04:11:48.317 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-d7d59826-eb26-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-d7d59826-eb26-11ee-95d0-87774f69a715.target. 2024-03-26T04:11:48.318 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-d7d59826-eb26-11ee-95d0-87774f69a715.target → /etc/systemd/system/ceph-d7d59826-eb26-11ee-95d0-87774f69a715.target. 2024-03-26T04:11:48.543 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-d7d59826-eb26-11ee-95d0-87774f69a715@mon.a 2024-03-26T04:11:48.543 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Failed to reset failed state of unit ceph-d7d59826-eb26-11ee-95d0-87774f69a715@mon.a.service: Unit ceph-d7d59826-eb26-11ee-95d0-87774f69a715@mon.a.service not loaded. 2024-03-26T04:11:48.698 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-d7d59826-eb26-11ee-95d0-87774f69a715.target.wants/ceph-d7d59826-eb26-11ee-95d0-87774f69a715@mon.a.service → /etc/systemd/system/ceph-d7d59826-eb26-11ee-95d0-87774f69a715@.service. 2024-03-26T04:11:49.624 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T04:11:49.624 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout disabled 2024-03-26T04:11:49.634 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T04:11:49.634 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-03-26T04:11:49.634 INFO:teuthology.orchestra.run.smithi016.stdout:firewalld.service is not enabled 2024-03-26T04:11:49.634 INFO:teuthology.orchestra.run.smithi016.stdout:Not possible to enable service . firewalld.service is not available 2024-03-26T04:11:49.635 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mon to start... 2024-03-26T04:11:49.635 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mon... 2024-03-26T04:11:50.708 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout cluster: 2024-03-26T04:11:50.708 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout id: d7d59826-eb26-11ee-95d0-87774f69a715 2024-03-26T04:11:50.709 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-26T04:11:50.709 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-03-26T04:11:50.709 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout services: 2024-03-26T04:11:50.709 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.517108s) 2024-03-26T04:11:50.709 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-26T04:11:50.709 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-26T04:11:50.709 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-03-26T04:11:50.709 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout data: 2024-03-26T04:11:50.709 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-26T04:11:50.709 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-26T04:11:50.709 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-26T04:11:50.709 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout pgs: 2024-03-26T04:11:50.710 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-03-26T04:11:50.710 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:50.047+0000 7fdf42c40700 1 Processor -- start 2024-03-26T04:11:50.710 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:50.047+0000 7fdf42c40700 1 -- start start 2024-03-26T04:11:50.710 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:50.047+0000 7fdf42c40700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdf3c078de0 con 0x7fdf3c106a70 2024-03-26T04:11:50.710 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:50.048+0000 7fdf409dc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fdf3c106a70 0x7fdf3c108e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52058/0 (socket says 172.21.15.16:52058) 2024-03-26T04:11:50.710 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:50.048+0000 7fdf409dc700 1 -- 172.21.15.16:0/2909841688 learned_addr learned my addr 172.21.15.16:0/2909841688 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:11:50.710 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:50.048+0000 7fdf3b7fe700 1 -- 172.21.15.16:0/2909841688 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 997457231 0 0) 0x7fdf3c078de0 con 0x7fdf3c106a70 2024-03-26T04:11:50.710 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:50.048+0000 7fdf3b7fe700 1 -- 172.21.15.16:0/2909841688 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf2c0035b0 con 0x7fdf3c106a70 2024-03-26T04:11:50.710 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:50.048+0000 7fdf3b7fe700 1 -- 172.21.15.16:0/2909841688 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2856014192 0 0) 0x7fdf2c0035b0 con 0x7fdf3c106a70 2024-03-26T04:11:50.710 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:50.048+0000 7fdf3b7fe700 1 -- 172.21.15.16:0/2909841688 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf3c10b590 con 0x7fdf3c106a70 2024-03-26T04:11:50.711 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:50.048+0000 7fdf3b7fe700 1 -- 172.21.15.16:0/2909841688 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7fdf30001580 con 0x7fdf3c106a70 2024-03-26T04:11:50.711 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:50.049+0000 7fdf3b7fe700 1 -- 172.21.15.16:0/2909841688 <== mon.0 v1:172.21.15.16:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdf300018e0 con 0x7fdf3c106a70 2024-03-26T04:11:50.711 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:50.049+0000 7fdf3b7fe700 1 -- 172.21.15.16:0/2909841688 <== mon.0 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7fdf30001c20 con 0x7fdf3c106a70 2024-03-26T04:11:50.711 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:50.049+0000 7fdf42c40700 1 -- 172.21.15.16:0/2909841688 >> v1:172.21.15.16:6789/0 conn(0x7fdf3c106a70 legacy=0x7fdf3c108e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:11:50.711 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:50.049+0000 7fdf42c40700 1 -- 172.21.15.16:0/2909841688 shutdown_connections 2024-03-26T04:11:50.711 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:50.049+0000 7fdf42c40700 1 -- 172.21.15.16:0/2909841688 >> 172.21.15.16:0/2909841688 conn(0x7fdf3c1003c0 msgr2=0x7fdf3c102800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:11:50.711 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:50.049+0000 7fdf42c40700 1 -- 172.21.15.16:0/2909841688 shutdown_connections 2024-03-26T04:11:50.711 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:50.049+0000 7fdf42c40700 1 -- 172.21.15.16:0/2909841688 wait complete. 2024-03-26T04:11:50.711 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:50.050+0000 7fdf42c40700 1 Processor -- start 2024-03-26T04:11:50.711 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:50.050+0000 7fdf42c40700 1 -- start start 2024-03-26T04:11:50.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:50.050+0000 7fdf42c40700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdf3c199680 con 0x7fdf3c106a70 2024-03-26T04:11:50.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:50.050+0000 7fdf409dc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fdf3c106a70 0x7fdf3c198f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52072/0 (socket says 172.21.15.16:52072) 2024-03-26T04:11:50.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:50.050+0000 7fdf409dc700 1 -- 172.21.15.16:0/376578572 learned_addr learned my addr 172.21.15.16:0/376578572 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:11:50.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:50.050+0000 7fdf39ffb700 1 -- 172.21.15.16:0/376578572 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2984253681 0 0) 0x7fdf3c199680 con 0x7fdf3c106a70 2024-03-26T04:11:50.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:50.050+0000 7fdf39ffb700 1 -- 172.21.15.16:0/376578572 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf2c0031f0 con 0x7fdf3c106a70 2024-03-26T04:11:50.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:50.050+0000 7fdf39ffb700 1 -- 172.21.15.16:0/376578572 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 544497515 0 0) 0x7fdf2c0031f0 con 0x7fdf3c106a70 2024-03-26T04:11:50.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:50.050+0000 7fdf39ffb700 1 -- 172.21.15.16:0/376578572 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf3c199680 con 0x7fdf3c106a70 2024-03-26T04:11:50.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:50.050+0000 7fdf39ffb700 1 -- 172.21.15.16:0/376578572 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7fdf30001480 con 0x7fdf3c106a70 2024-03-26T04:11:50.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:50.051+0000 7fdf39ffb700 1 -- 172.21.15.16:0/376578572 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1735184384 0 0) 0x7fdf3c199680 con 0x7fdf3c106a70 2024-03-26T04:11:50.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:50.051+0000 7fdf39ffb700 1 -- 172.21.15.16:0/376578572 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf3c199810 con 0x7fdf3c106a70 2024-03-26T04:11:50.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:50.051+0000 7fdf42c40700 1 -- 172.21.15.16:0/376578572 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf3c199ac0 con 0x7fdf3c106a70 2024-03-26T04:11:50.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:50.051+0000 7fdf39ffb700 1 -- 172.21.15.16:0/376578572 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdf30004c80 con 0x7fdf3c106a70 2024-03-26T04:11:50.712 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:50.051+0000 7fdf39ffb700 1 -- 172.21.15.16:0/376578572 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7fdf30004f80 con 0x7fdf3c106a70 2024-03-26T04:11:50.713 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:50.051+0000 7fdf39ffb700 1 -- 172.21.15.16:0/376578572 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fdf30005570 con 0x7fdf3c106a70 2024-03-26T04:11:50.713 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:50.051+0000 7fdf42c40700 1 -- 172.21.15.16:0/376578572 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf3c19d610 con 0x7fdf3c106a70 2024-03-26T04:11:50.713 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:50.051+0000 7fdf39ffb700 1 -- 172.21.15.16:0/376578572 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 1322898288 0 0) 0x7fdf30005d90 con 0x7fdf3c106a70 2024-03-26T04:11:50.713 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:50.052+0000 7fdf42c40700 1 -- 172.21.15.16:0/376578572 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf3c192be0 con 0x7fdf3c106a70 2024-03-26T04:11:50.713 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:50.054+0000 7fdf39ffb700 1 -- 172.21.15.16:0/376578572 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fdf30006930 con 0x7fdf3c106a70 2024-03-26T04:11:50.713 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:50.127+0000 7fdf42c40700 1 -- 172.21.15.16:0/376578572 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fdf3c10a660 con 0x7fdf3c106a70 2024-03-26T04:11:50.713 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:50.127+0000 7fdf39ffb700 1 -- 172.21.15.16:0/376578572 <== mon.0 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (unknown 1155462804 0 2594852751) 0x7fdf30006b50 con 0x7fdf3c106a70 2024-03-26T04:11:50.713 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:50.128+0000 7fdf42c40700 1 -- 172.21.15.16:0/376578572 >> v1:172.21.15.16:6789/0 conn(0x7fdf3c106a70 legacy=0x7fdf3c198f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:11:50.713 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:50.128+0000 7fdf42c40700 1 -- 172.21.15.16:0/376578572 shutdown_connections 2024-03-26T04:11:50.713 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:50.128+0000 7fdf42c40700 1 -- 172.21.15.16:0/376578572 >> 172.21.15.16:0/376578572 conn(0x7fdf3c1003c0 msgr2=0x7fdf3c107730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:11:50.713 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:50.128+0000 7fdf42c40700 1 -- 172.21.15.16:0/376578572 shutdown_connections 2024-03-26T04:11:50.713 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:50.128+0000 7fdf42c40700 1 -- 172.21.15.16:0/376578572 wait complete. 2024-03-26T04:11:50.713 INFO:teuthology.orchestra.run.smithi016.stdout:mon is available 2024-03-26T04:11:50.714 INFO:teuthology.orchestra.run.smithi016.stdout:Assimilating anything we can from ceph.conf... 2024-03-26T04:11:51.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:51 smithi016 ceph-mon[28729]: from='client.? v1:172.21.15.16:0/3048091295' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-26T04:11:52.481 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-03-26T04:11:52.481 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [global] 2024-03-26T04:11:52.481 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout fsid = d7d59826-eb26-11ee-95d0-87774f69a715 2024-03-26T04:11:52.481 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_host = [v1:172.21.15.16:6789] 2024-03-26T04:11:52.481 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-26T04:11:52.481 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-26T04:11:52.481 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-26T04:11:52.481 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-26T04:11:52.481 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-03-26T04:11:52.482 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-26T04:11:52.482 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-26T04:11:52.482 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-03-26T04:11:52.482 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [osd] 2024-03-26T04:11:52.482 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-26T04:11:52.482 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-26T04:11:52.482 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:51.113+0000 7fa2f80dc700 1 Processor -- start 2024-03-26T04:11:52.482 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:51.113+0000 7fa2f80dc700 1 -- start start 2024-03-26T04:11:52.482 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:51.113+0000 7fa2f80dc700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa2f007a380 con 0x7fa2f007b780 2024-03-26T04:11:52.482 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:51.113+0000 7fa2f5e78700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa2f007b780 0x7fa2f0079be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52078/0 (socket says 172.21.15.16:52078) 2024-03-26T04:11:52.482 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:51.113+0000 7fa2f5e78700 1 -- 172.21.15.16:0/2843842990 learned_addr learned my addr 172.21.15.16:0/2843842990 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:11:52.482 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:51.113+0000 7fa2f4e76700 1 -- 172.21.15.16:0/2843842990 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1234714984 0 0) 0x7fa2f007a380 con 0x7fa2f007b780 2024-03-26T04:11:52.483 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:51.113+0000 7fa2f4e76700 1 -- 172.21.15.16:0/2843842990 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2dc0035b0 con 0x7fa2f007b780 2024-03-26T04:11:52.483 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:51.114+0000 7fa2f4e76700 1 -- 172.21.15.16:0/2843842990 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4128029747 0 0) 0x7fa2dc0035b0 con 0x7fa2f007b780 2024-03-26T04:11:52.483 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:51.114+0000 7fa2f4e76700 1 -- 172.21.15.16:0/2843842990 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f007a510 con 0x7fa2f007b780 2024-03-26T04:11:52.483 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:51.114+0000 7fa2f4e76700 1 -- 172.21.15.16:0/2843842990 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7fa2ec001580 con 0x7fa2f007b780 2024-03-26T04:11:52.483 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:51.114+0000 7fa2f4e76700 1 -- 172.21.15.16:0/2843842990 <== mon.0 v1:172.21.15.16:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2ec0018e0 con 0x7fa2f007b780 2024-03-26T04:11:52.483 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:51.114+0000 7fa2f4e76700 1 -- 172.21.15.16:0/2843842990 <== mon.0 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7fa2ec001c20 con 0x7fa2f007b780 2024-03-26T04:11:52.483 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:51.114+0000 7fa2f80dc700 1 -- 172.21.15.16:0/2843842990 >> v1:172.21.15.16:6789/0 conn(0x7fa2f007b780 legacy=0x7fa2f0079be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:11:52.483 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:51.114+0000 7fa2f80dc700 1 -- 172.21.15.16:0/2843842990 shutdown_connections 2024-03-26T04:11:52.483 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:51.114+0000 7fa2f80dc700 1 -- 172.21.15.16:0/2843842990 >> 172.21.15.16:0/2843842990 conn(0x7fa2f01014e0 msgr2=0x7fa2f0103900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:11:52.483 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:51.114+0000 7fa2f80dc700 1 -- 172.21.15.16:0/2843842990 shutdown_connections 2024-03-26T04:11:52.483 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:51.114+0000 7fa2f80dc700 1 -- 172.21.15.16:0/2843842990 wait complete. 2024-03-26T04:11:52.484 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:51.115+0000 7fa2f80dc700 1 Processor -- start 2024-03-26T04:11:52.484 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:51.115+0000 7fa2f80dc700 1 -- start start 2024-03-26T04:11:52.484 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:51.115+0000 7fa2f80dc700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa2f019b900 con 0x7fa2f007b780 2024-03-26T04:11:52.484 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:51.115+0000 7fa2f5e78700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa2f007b780 0x7fa2f019b1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52080/0 (socket says 172.21.15.16:52080) 2024-03-26T04:11:52.484 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:51.115+0000 7fa2f5e78700 1 -- 172.21.15.16:0/3048091295 learned_addr learned my addr 172.21.15.16:0/3048091295 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:11:52.484 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:51.115+0000 7fa2e6ffd700 1 -- 172.21.15.16:0/3048091295 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3693118196 0 0) 0x7fa2f019b900 con 0x7fa2f007b780 2024-03-26T04:11:52.484 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:51.115+0000 7fa2e6ffd700 1 -- 172.21.15.16:0/3048091295 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2dc0031f0 con 0x7fa2f007b780 2024-03-26T04:11:52.484 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:51.116+0000 7fa2e6ffd700 1 -- 172.21.15.16:0/3048091295 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4178703883 0 0) 0x7fa2dc0031f0 con 0x7fa2f007b780 2024-03-26T04:11:52.484 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:51.116+0000 7fa2e6ffd700 1 -- 172.21.15.16:0/3048091295 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2f019b900 con 0x7fa2f007b780 2024-03-26T04:11:52.484 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:51.116+0000 7fa2e6ffd700 1 -- 172.21.15.16:0/3048091295 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7fa2ec001480 con 0x7fa2f007b780 2024-03-26T04:11:52.484 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:51.116+0000 7fa2e6ffd700 1 -- 172.21.15.16:0/3048091295 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3445003051 0 0) 0x7fa2f019b900 con 0x7fa2f007b780 2024-03-26T04:11:52.485 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:51.116+0000 7fa2e6ffd700 1 -- 172.21.15.16:0/3048091295 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f019ba90 con 0x7fa2f007b780 2024-03-26T04:11:52.485 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:51.116+0000 7fa2f80dc700 1 -- 172.21.15.16:0/3048091295 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2f019bd40 con 0x7fa2f007b780 2024-03-26T04:11:52.485 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:51.116+0000 7fa2f80dc700 1 -- 172.21.15.16:0/3048091295 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2f019f890 con 0x7fa2f007b780 2024-03-26T04:11:52.485 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:51.116+0000 7fa2e6ffd700 1 -- 172.21.15.16:0/3048091295 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2ec004c80 con 0x7fa2f007b780 2024-03-26T04:11:52.485 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:51.116+0000 7fa2e6ffd700 1 -- 172.21.15.16:0/3048091295 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7fa2ec004df0 con 0x7fa2f007b780 2024-03-26T04:11:52.485 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:51.116+0000 7fa2e6ffd700 1 -- 172.21.15.16:0/3048091295 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fa2ec0052e0 con 0x7fa2f007b780 2024-03-26T04:11:52.485 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:51.116+0000 7fa2e6ffd700 1 -- 172.21.15.16:0/3048091295 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 1322898288 0 0) 0x7fa2ec0068e0 con 0x7fa2f007b780 2024-03-26T04:11:52.485 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:51.117+0000 7fa2f80dc700 1 -- 172.21.15.16:0/3048091295 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2d4005150 con 0x7fa2f007b780 2024-03-26T04:11:52.485 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:51.119+0000 7fa2e6ffd700 1 -- 172.21.15.16:0/3048091295 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fa2ec006af0 con 0x7fa2f007b780 2024-03-26T04:11:52.485 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:51.191+0000 7fa2f80dc700 1 -- 172.21.15.16:0/3048091295 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fa2d4005980 con 0x7fa2f007b780 2024-03-26T04:11:52.485 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:51.194+0000 7fa2e6ffd700 1 -- 172.21.15.16:0/3048091295 <== mon.0 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+317 (unknown 1213389831 0 487264548) 0x7fa2ec0057d0 con 0x7fa2f007b780 2024-03-26T04:11:52.486 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:51.194+0000 7fa2e6ffd700 1 -- 172.21.15.16:0/3048091295 <== mon.0 v1:172.21.15.16:6789/0 11 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 848163187 0 0) 0x7fa2ec005e60 con 0x7fa2f007b780 2024-03-26T04:11:52.486 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:51.195+0000 7fa2f80dc700 1 -- 172.21.15.16:0/3048091295 >> v1:172.21.15.16:6789/0 conn(0x7fa2f007b780 legacy=0x7fa2f019b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:11:52.486 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:51.195+0000 7fa2f80dc700 1 -- 172.21.15.16:0/3048091295 shutdown_connections 2024-03-26T04:11:52.486 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:51.195+0000 7fa2f80dc700 1 -- 172.21.15.16:0/3048091295 >> 172.21.15.16:0/3048091295 conn(0x7fa2f01014e0 msgr2=0x7fa2f010ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:11:52.486 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:51.195+0000 7fa2f80dc700 1 -- 172.21.15.16:0/3048091295 shutdown_connections 2024-03-26T04:11:52.486 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:51.195+0000 7fa2f80dc700 1 -- 172.21.15.16:0/3048091295 wait complete. 2024-03-26T04:11:52.486 INFO:teuthology.orchestra.run.smithi016.stdout:Generating new minimal ceph.conf... 2024-03-26T04:11:53.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:53 smithi016 ceph-mon[28729]: from='client.? v1:172.21.15.16:0/793524963' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:11:53.542 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:52.918+0000 7f89f8e8c700 1 Processor -- start 2024-03-26T04:11:53.543 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:52.919+0000 7f89f8e8c700 1 -- start start 2024-03-26T04:11:53.543 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:52.919+0000 7f89f8e8c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f89f410a340 con 0x7f89f4105f20 2024-03-26T04:11:53.543 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:52.919+0000 7f89f259c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f89f4105f20 0x7f89f4106340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52086/0 (socket says 172.21.15.16:52086) 2024-03-26T04:11:53.543 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:52.919+0000 7f89f259c700 1 -- 172.21.15.16:0/2933658515 learned_addr learned my addr 172.21.15.16:0/2933658515 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:11:53.543 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:52.919+0000 7f89f159a700 1 -- 172.21.15.16:0/2933658515 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2275873868 0 0) 0x7f89f410a340 con 0x7f89f4105f20 2024-03-26T04:11:53.543 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:52.919+0000 7f89f159a700 1 -- 172.21.15.16:0/2933658515 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89dc0035b0 con 0x7f89f4105f20 2024-03-26T04:11:53.543 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:52.920+0000 7f89f159a700 1 -- 172.21.15.16:0/2933658515 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 881100878 0 0) 0x7f89dc0035b0 con 0x7f89f4105f20 2024-03-26T04:11:53.543 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:52.920+0000 7f89f159a700 1 -- 172.21.15.16:0/2933658515 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f410b4e0 con 0x7f89f4105f20 2024-03-26T04:11:53.543 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:52.920+0000 7f89f159a700 1 -- 172.21.15.16:0/2933658515 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f89e8001580 con 0x7f89f4105f20 2024-03-26T04:11:53.543 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:52.920+0000 7f89f159a700 1 -- 172.21.15.16:0/2933658515 <== mon.0 v1:172.21.15.16:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 848163187 0 0) 0x7f89e8001ce0 con 0x7f89f4105f20 2024-03-26T04:11:53.543 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:52.920+0000 7f89f159a700 1 -- 172.21.15.16:0/2933658515 <== mon.0 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f89e8005960 con 0x7f89f4105f20 2024-03-26T04:11:53.543 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:52.920+0000 7f89f8e8c700 1 -- 172.21.15.16:0/2933658515 >> v1:172.21.15.16:6789/0 conn(0x7f89f4105f20 legacy=0x7f89f4106340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:11:53.544 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:52.920+0000 7f89f8e8c700 1 -- 172.21.15.16:0/2933658515 shutdown_connections 2024-03-26T04:11:53.544 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:52.920+0000 7f89f8e8c700 1 -- 172.21.15.16:0/2933658515 >> 172.21.15.16:0/2933658515 conn(0x7f89f41014e0 msgr2=0x7f89f4103900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:11:53.544 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:52.920+0000 7f89f8e8c700 1 -- 172.21.15.16:0/2933658515 shutdown_connections 2024-03-26T04:11:53.544 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:52.921+0000 7f89f8e8c700 1 -- 172.21.15.16:0/2933658515 wait complete. 2024-03-26T04:11:53.544 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:52.921+0000 7f89f8e8c700 1 Processor -- start 2024-03-26T04:11:53.544 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:52.921+0000 7f89f8e8c700 1 -- start start 2024-03-26T04:11:53.544 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:52.921+0000 7f89f8e8c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f89f4199ea0 con 0x7f89f4105f20 2024-03-26T04:11:53.544 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:52.922+0000 7f89f259c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f89f4105f20 0x7f89f4199790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:52092/0 (socket says 172.21.15.16:52092) 2024-03-26T04:11:53.544 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:52.922+0000 7f89f259c700 1 -- 172.21.15.16:0/793524963 learned_addr learned my addr 172.21.15.16:0/793524963 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:11:53.544 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:52.922+0000 7f89e37fe700 1 -- 172.21.15.16:0/793524963 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1004875446 0 0) 0x7f89f4199ea0 con 0x7f89f4105f20 2024-03-26T04:11:53.544 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:52.922+0000 7f89e37fe700 1 -- 172.21.15.16:0/793524963 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89dc0055b0 con 0x7f89f4105f20 2024-03-26T04:11:53.544 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:52.922+0000 7f89e37fe700 1 -- 172.21.15.16:0/793524963 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2542556904 0 0) 0x7f89dc0055b0 con 0x7f89f4105f20 2024-03-26T04:11:53.544 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:52.922+0000 7f89e37fe700 1 -- 172.21.15.16:0/793524963 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89f4199ea0 con 0x7f89f4105f20 2024-03-26T04:11:53.544 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:52.922+0000 7f89e37fe700 1 -- 172.21.15.16:0/793524963 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f89e8004870 con 0x7f89f4105f20 2024-03-26T04:11:53.545 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:52.922+0000 7f89e37fe700 1 -- 172.21.15.16:0/793524963 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 39430634 0 0) 0x7f89f4199ea0 con 0x7f89f4105f20 2024-03-26T04:11:53.545 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:52.922+0000 7f89e37fe700 1 -- 172.21.15.16:0/793524963 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f419a030 con 0x7f89f4105f20 2024-03-26T04:11:53.545 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:52.922+0000 7f89f8e8c700 1 -- 172.21.15.16:0/793524963 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89f419a2e0 con 0x7f89f4105f20 2024-03-26T04:11:53.545 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:52.922+0000 7f89f8e8c700 1 -- 172.21.15.16:0/793524963 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f89f419de30 con 0x7f89f4105f20 2024-03-26T04:11:53.545 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:52.922+0000 7f89e37fe700 1 -- 172.21.15.16:0/793524963 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 848163187 0 0) 0x7f89e8004a40 con 0x7f89f4105f20 2024-03-26T04:11:53.545 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:52.922+0000 7f89e37fe700 1 -- 172.21.15.16:0/793524963 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f89e8006060 con 0x7f89f4105f20 2024-03-26T04:11:53.545 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:52.922+0000 7f89e37fe700 1 -- 172.21.15.16:0/793524963 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f89e8006630 con 0x7f89f4105f20 2024-03-26T04:11:53.545 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:52.923+0000 7f89e37fe700 1 -- 172.21.15.16:0/793524963 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 1322898288 0 0) 0x7f89e8007c60 con 0x7f89f4105f20 2024-03-26T04:11:53.545 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:52.923+0000 7f89f8e8c700 1 -- 172.21.15.16:0/793524963 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89d4005150 con 0x7f89f4105f20 2024-03-26T04:11:53.545 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:52.925+0000 7f89e37fe700 1 -- 172.21.15.16:0/793524963 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f89e8005640 con 0x7f89f4105f20 2024-03-26T04:11:53.545 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:52.996+0000 7f89f8e8c700 1 -- 172.21.15.16:0/793524963 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f89d4005400 con 0x7f89f4105f20 2024-03-26T04:11:53.545 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:52.996+0000 7f89e37fe700 1 -- 172.21.15.16:0/793524963 <== mon.0 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+147 (unknown 1452402520 0 1139779224) 0x7f89e80057e0 con 0x7f89f4105f20 2024-03-26T04:11:53.545 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:52.997+0000 7f89f8e8c700 1 -- 172.21.15.16:0/793524963 >> v1:172.21.15.16:6789/0 conn(0x7f89f4105f20 legacy=0x7f89f4199790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:11:53.546 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:52.997+0000 7f89f8e8c700 1 -- 172.21.15.16:0/793524963 shutdown_connections 2024-03-26T04:11:53.546 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:52.997+0000 7f89f8e8c700 1 -- 172.21.15.16:0/793524963 >> 172.21.15.16:0/793524963 conn(0x7f89f41014e0 msgr2=0x7f89f41038d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:11:53.546 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:52.997+0000 7f89f8e8c700 1 -- 172.21.15.16:0/793524963 shutdown_connections 2024-03-26T04:11:53.546 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:52.997+0000 7f89f8e8c700 1 -- 172.21.15.16:0/793524963 wait complete. 2024-03-26T04:11:53.546 INFO:teuthology.orchestra.run.smithi016.stdout:Restarting the monitor... 2024-03-26T04:11:53.815 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:53 smithi016 systemd[1]: Stopping Ceph mon.a for d7d59826-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:11:54.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:53 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mon-a[28725]: 2024-03-26T04:11:53.821+0000 7f83dd9ce700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:11:54.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:53 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mon-a[28725]: 2024-03-26T04:11:53.821+0000 7f83dd9ce700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-26T04:11:54.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:54 smithi016 podman[28973]: 2024-03-26 04:11:54.026574279 +0000 UTC m=+0.317851171 container died 6b79afa9fd35027c97a8c9d577976f288516e890c1fd1fb29bae8f4060ed135e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mon-a, RELEASE=squid-5e70124, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-26T04:11:54.647 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:54 smithi016 podman[28973]: 2024-03-26 04:11:54.413723982 +0000 UTC m=+0.705000866 container cleanup 6b79afa9fd35027c97a8c9d577976f288516e890c1fd1fb29bae8f4060ed135e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mon-a, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, org.label-schema.license=GPLv2) 2024-03-26T04:11:54.648 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:54 smithi016 bash[28973]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mon-a 2024-03-26T04:11:54.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:54 smithi016 podman[28987]: 2024-03-26 04:11:54.647773768 +0000 UTC m=+0.618961327 container remove 6b79afa9fd35027c97a8c9d577976f288516e890c1fd1fb29bae8f4060ed135e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mon-a, RELEASE=squid-5e70124, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T04:11:54.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:54 smithi016 systemd[1]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-26T04:11:54.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:54 smithi016 systemd[1]: Stopped Ceph mon.a for d7d59826-eb26-11ee-95d0-87774f69a715. 2024-03-26T04:11:54.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:54 smithi016 systemd[1]: Starting Ceph mon.a for d7d59826-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:11:55.276 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:54 smithi016 podman[29082]: 2024-03-26 04:11:54.990614735 +0000 UTC m=+0.110202093 container create 4eb1141880f76e78f97440d7f4a860c981dc4f24726fca4ecbbccdaa2b3b8cea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mon-a, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-26T04:11:55.276 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:54 smithi016 podman[29082]: 2024-03-26 04:11:54.897705437 +0000 UTC m=+0.017292810 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a 2024-03-26T04:11:55.277 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 podman[29082]: 2024-03-26 04:11:55.193226659 +0000 UTC m=+0.312814019 container init 4eb1141880f76e78f97440d7f4a860c981dc4f24726fca4ecbbccdaa2b3b8cea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mon-a, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-26T04:11:55.277 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 podman[29082]: 2024-03-26 04:11:55.19627367 +0000 UTC m=+0.315861022 container start 4eb1141880f76e78f97440d7f4a860c981dc4f24726fca4ecbbccdaa2b3b8cea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T04:11:55.277 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T04:11:55.277 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev), process ceph-mon, pid 2 2024-03-26T04:11:55.277 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: pidfile_write: ignore empty --pid-file 2024-03-26T04:11:55.277 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: load: jerasure load: lrc 2024-03-26T04:11:55.277 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: RocksDB version: 7.9.2 2024-03-26T04:11:55.277 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Git sha 0 2024-03-26T04:11:55.277 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Compile date 2024-03-25 16:14:20 2024-03-26T04:11:55.277 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: DB SUMMARY 2024-03-26T04:11:55.278 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: DB Session ID: UNRUU1E6W2LA4GXF3T99 2024-03-26T04:11:55.278 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: CURRENT file: CURRENT 2024-03-26T04:11:55.278 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T04:11:55.278 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-26T04:11:55.278 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-26T04:11:55.278 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 85542 ; 2024-03-26T04:11:55.278 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.error_if_exists: 0 2024-03-26T04:11:55.278 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.create_if_missing: 0 2024-03-26T04:11:55.278 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.paranoid_checks: 1 2024-03-26T04:11:55.278 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-26T04:11:55.278 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T04:11:55.278 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-26T04:11:55.279 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.env: 0x5637ae532ae0 2024-03-26T04:11:55.279 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.fs: PosixFileSystem 2024-03-26T04:11:55.279 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.info_log: 0x5637b0b3d940 2024-03-26T04:11:55.279 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T04:11:55.279 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.statistics: (nil) 2024-03-26T04:11:55.279 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.use_fsync: 0 2024-03-26T04:11:55.279 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.max_log_file_size: 0 2024-03-26T04:11:55.279 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T04:11:55.279 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T04:11:55.279 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T04:11:55.279 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T04:11:55.280 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.allow_fallocate: 1 2024-03-26T04:11:55.280 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T04:11:55.280 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T04:11:55.280 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.use_direct_reads: 0 2024-03-26T04:11:55.280 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T04:11:55.280 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T04:11:55.280 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.db_log_dir: 2024-03-26T04:11:55.280 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.wal_dir: 2024-03-26T04:11:55.280 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T04:11:55.280 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T04:11:55.280 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T04:11:55.280 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T04:11:55.280 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T04:11:55.280 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T04:11:55.281 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T04:11:55.281 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T04:11:55.281 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.write_buffer_manager: 0x5637b0b101e0 2024-03-26T04:11:55.281 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T04:11:55.281 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T04:11:55.281 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T04:11:55.281 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T04:11:55.281 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T04:11:55.281 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T04:11:55.282 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T04:11:55.283 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T04:11:55.283 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.unordered_write: 0 2024-03-26T04:11:55.283 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T04:11:55.283 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T04:11:55.283 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T04:11:55.283 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T04:11:55.283 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.row_cache: None 2024-03-26T04:11:55.283 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.wal_filter: None 2024-03-26T04:11:55.283 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T04:11:55.283 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T04:11:55.284 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.two_write_queues: 0 2024-03-26T04:11:55.284 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T04:11:55.284 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.wal_compression: 0 2024-03-26T04:11:55.284 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.atomic_flush: 0 2024-03-26T04:11:55.284 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T04:11:55.284 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T04:11:55.284 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T04:11:55.284 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.log_readahead_size: 0 2024-03-26T04:11:55.284 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T04:11:55.284 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T04:11:55.284 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T04:11:55.284 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T04:11:55.284 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T04:11:55.284 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T04:11:55.285 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T04:11:55.285 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.max_background_jobs: 2 2024-03-26T04:11:55.285 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.max_background_compactions: -1 2024-03-26T04:11:55.285 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.max_subcompactions: 1 2024-03-26T04:11:55.285 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T04:11:55.285 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T04:11:55.285 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T04:11:55.285 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T04:11:55.285 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T04:11:55.285 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T04:11:55.285 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T04:11:55.285 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T04:11:55.285 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.max_open_files: -1 2024-03-26T04:11:55.285 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T04:11:55.286 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T04:11:55.286 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T04:11:55.286 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T04:11:55.286 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.max_background_flushes: -1 2024-03-26T04:11:55.286 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Compression algorithms supported: 2024-03-26T04:11:55.286 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T04:11:55.286 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: kZSTD supported: 0 2024-03-26T04:11:55.286 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: kXpressCompression supported: 0 2024-03-26T04:11:55.286 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T04:11:55.286 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: kZlibCompression supported: 1 2024-03-26T04:11:55.286 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: kSnappyCompression supported: 1 2024-03-26T04:11:55.286 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: kLZ4Compression supported: 1 2024-03-26T04:11:55.286 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: kBZip2Compression supported: 0 2024-03-26T04:11:55.287 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T04:11:55.287 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T04:11:55.287 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-26T04:11:55.287 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T04:11:55.287 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T04:11:55.287 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.merge_operator: 2024-03-26T04:11:55.287 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.compaction_filter: None 2024-03-26T04:11:55.287 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T04:11:55.287 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T04:11:55.287 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T04:11:55.287 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T04:11:55.287 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5637b0b3da40) 2024-03-26T04:11:55.288 INFO:journalctl@ceph.mon.a.smithi016.stdout: cache_index_and_filter_blocks: 1 2024-03-26T04:11:55.288 INFO:journalctl@ceph.mon.a.smithi016.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T04:11:55.288 INFO:journalctl@ceph.mon.a.smithi016.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T04:11:55.288 INFO:journalctl@ceph.mon.a.smithi016.stdout: pin_top_level_index_and_filter: 1 2024-03-26T04:11:55.288 INFO:journalctl@ceph.mon.a.smithi016.stdout: index_type: 0 2024-03-26T04:11:55.288 INFO:journalctl@ceph.mon.a.smithi016.stdout: data_block_index_type: 0 2024-03-26T04:11:55.288 INFO:journalctl@ceph.mon.a.smithi016.stdout: index_shortening: 1 2024-03-26T04:11:55.288 INFO:journalctl@ceph.mon.a.smithi016.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T04:11:55.288 INFO:journalctl@ceph.mon.a.smithi016.stdout: checksum: 4 2024-03-26T04:11:55.288 INFO:journalctl@ceph.mon.a.smithi016.stdout: no_block_cache: 0 2024-03-26T04:11:55.288 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache: 0x5637b0b41090 2024-03-26T04:11:55.288 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache_name: BinnedLRUCache 2024-03-26T04:11:55.288 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache_options: 2024-03-26T04:11:55.289 INFO:journalctl@ceph.mon.a.smithi016.stdout: capacity : 536870912 2024-03-26T04:11:55.289 INFO:journalctl@ceph.mon.a.smithi016.stdout: num_shard_bits : 4 2024-03-26T04:11:55.289 INFO:journalctl@ceph.mon.a.smithi016.stdout: strict_capacity_limit : 0 2024-03-26T04:11:55.289 INFO:journalctl@ceph.mon.a.smithi016.stdout: high_pri_pool_ratio: 0.000 2024-03-26T04:11:55.289 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_cache_compressed: (nil) 2024-03-26T04:11:55.289 INFO:journalctl@ceph.mon.a.smithi016.stdout: persistent_cache: (nil) 2024-03-26T04:11:55.289 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_size: 4096 2024-03-26T04:11:55.289 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_size_deviation: 10 2024-03-26T04:11:55.289 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_restart_interval: 16 2024-03-26T04:11:55.289 INFO:journalctl@ceph.mon.a.smithi016.stdout: index_block_restart_interval: 1 2024-03-26T04:11:55.289 INFO:journalctl@ceph.mon.a.smithi016.stdout: metadata_block_size: 4096 2024-03-26T04:11:55.289 INFO:journalctl@ceph.mon.a.smithi016.stdout: partition_filters: 0 2024-03-26T04:11:55.289 INFO:journalctl@ceph.mon.a.smithi016.stdout: use_delta_encoding: 1 2024-03-26T04:11:55.290 INFO:journalctl@ceph.mon.a.smithi016.stdout: filter_policy: bloomfilter 2024-03-26T04:11:55.290 INFO:journalctl@ceph.mon.a.smithi016.stdout: whole_key_filtering: 1 2024-03-26T04:11:55.290 INFO:journalctl@ceph.mon.a.smithi016.stdout: verify_compression: 0 2024-03-26T04:11:55.290 INFO:journalctl@ceph.mon.a.smithi016.stdout: read_amp_bytes_per_bit: 0 2024-03-26T04:11:55.290 INFO:journalctl@ceph.mon.a.smithi016.stdout: format_version: 5 2024-03-26T04:11:55.290 INFO:journalctl@ceph.mon.a.smithi016.stdout: enable_index_compression: 1 2024-03-26T04:11:55.290 INFO:journalctl@ceph.mon.a.smithi016.stdout: block_align: 0 2024-03-26T04:11:55.290 INFO:journalctl@ceph.mon.a.smithi016.stdout: max_auto_readahead_size: 262144 2024-03-26T04:11:55.290 INFO:journalctl@ceph.mon.a.smithi016.stdout: prepopulate_block_cache: 0 2024-03-26T04:11:55.290 INFO:journalctl@ceph.mon.a.smithi016.stdout: initial_auto_readahead_size: 8192 2024-03-26T04:11:55.290 INFO:journalctl@ceph.mon.a.smithi016.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T04:11:55.290 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T04:11:55.290 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T04:11:55.290 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.compression: NoCompression 2024-03-26T04:11:55.291 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T04:11:55.291 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T04:11:55.291 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T04:11:55.291 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.num_levels: 7 2024-03-26T04:11:55.291 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T04:11:55.291 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T04:11:55.291 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T04:11:55.291 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T04:11:55.291 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T04:11:55.291 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T04:11:55.291 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T04:11:55.291 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T04:11:55.291 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T04:11:55.292 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T04:11:55.292 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T04:11:55.292 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T04:11:55.292 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T04:11:55.292 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T04:11:55.292 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T04:11:55.292 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T04:11:55.292 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T04:11:55.292 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T04:11:55.292 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T04:11:55.292 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T04:11:55.292 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T04:11:55.292 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T04:11:55.293 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T04:11:55.293 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T04:11:55.293 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T04:11:55.293 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T04:11:55.293 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T04:11:55.293 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T04:11:55.293 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T04:11:55.293 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T04:11:55.293 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T04:11:55.293 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T04:11:55.293 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T04:11:55.293 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T04:11:55.293 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T04:11:55.294 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T04:11:55.294 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T04:11:55.294 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T04:11:55.294 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T04:11:55.294 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T04:11:55.294 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T04:11:55.294 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T04:11:55.294 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T04:11:55.294 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T04:11:55.294 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T04:11:55.294 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T04:11:55.294 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T04:11:55.295 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T04:11:55.295 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T04:11:55.295 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T04:11:55.295 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T04:11:55.295 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T04:11:55.295 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T04:11:55.295 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.table_properties_collectors: 2024-03-26T04:11:55.295 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.inplace_update_support: 0 2024-03-26T04:11:55.295 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T04:11:55.295 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T04:11:55.295 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T04:11:55.295 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T04:11:55.295 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.bloom_locality: 0 2024-03-26T04:11:55.296 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.max_successive_merges: 0 2024-03-26T04:11:55.296 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T04:11:55.296 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T04:11:55.296 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T04:11:55.296 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T04:11:55.296 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.ttl: 2592000 2024-03-26T04:11:55.296 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T04:11:55.296 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T04:11:55.296 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T04:11:55.296 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.enable_blob_files: false 2024-03-26T04:11:55.296 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.min_blob_size: 0 2024-03-26T04:11:55.296 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T04:11:55.297 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T04:11:55.297 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T04:11:55.297 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T04:11:55.297 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T04:11:55.297 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T04:11:55.297 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T04:11:55.297 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T04:11:55.297 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-26T04:11:55.297 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-26T04:11:55.297 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: f5ee864f-dcae-4720-b388-200266406b40 2024-03-26T04:11:55.297 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711426315225150, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-26T04:11:55.297 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-26T04:11:55.298 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711426315226022, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 81876, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 264, "table_properties": {"data_size": 80075, "index_size": 211, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 10703, "raw_average_key_size": 47, "raw_value_size": 74050, "raw_average_value_size": 332, "num_data_blocks": 10, "num_entries": 223, "num_filter_entries": 223, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711426315, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "f5ee864f-dcae-4720-b388-200266406b40", "db_session_id": "UNRUU1E6W2LA4GXF3T99", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-26T04:11:55.298 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711426315226138, "job": 1, "event": "recovery_finished"} 2024-03-26T04:11:55.298 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-26T04:11:55.298 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T04:11:55.298 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5637b0c32000 2024-03-26T04:11:55.298 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: DB pointer 0x5637b0c1c000 2024-03-26T04:11:55.298 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T04:11:55.298 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T04:11:55.298 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** DB Stats ** 2024-03-26T04:11:55.298 INFO:journalctl@ceph.mon.a.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T04:11:55.298 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T04:11:55.298 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T04:11:55.299 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T04:11:55.299 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T04:11:55.299 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T04:11:55.299 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T04:11:55.299 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-03-26T04:11:55.299 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** Compaction Stats [default] ** 2024-03-26T04:11:55.299 INFO:journalctl@ceph.mon.a.smithi016.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T04:11:55.299 INFO:journalctl@ceph.mon.a.smithi016.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T04:11:55.299 INFO:journalctl@ceph.mon.a.smithi016.stdout: L0 2/0 81.76 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 135.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T04:11:55.299 INFO:journalctl@ceph.mon.a.smithi016.stdout: Sum 2/0 81.76 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 135.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T04:11:55.299 INFO:journalctl@ceph.mon.a.smithi016.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 135.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T04:11:55.299 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-03-26T04:11:55.300 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** Compaction Stats [default] ** 2024-03-26T04:11:55.300 INFO:journalctl@ceph.mon.a.smithi016.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T04:11:55.300 INFO:journalctl@ceph.mon.a.smithi016.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T04:11:55.300 INFO:journalctl@ceph.mon.a.smithi016.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 135.1 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-26T04:11:55.300 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-03-26T04:11:55.300 INFO:journalctl@ceph.mon.a.smithi016.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T04:11:55.300 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-03-26T04:11:55.300 INFO:journalctl@ceph.mon.a.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T04:11:55.300 INFO:journalctl@ceph.mon.a.smithi016.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-26T04:11:55.300 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T04:11:55.301 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T04:11:55.301 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T04:11:55.301 INFO:journalctl@ceph.mon.a.smithi016.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T04:11:55.301 INFO:journalctl@ceph.mon.a.smithi016.stdout: Cumulative compaction: 0.00 GB write, 22.25 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T04:11:55.301 INFO:journalctl@ceph.mon.a.smithi016.stdout: Interval compaction: 0.00 GB write, 22.25 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T04:11:55.301 INFO:journalctl@ceph.mon.a.smithi016.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T04:11:55.301 INFO:journalctl@ceph.mon.a.smithi016.stdout: Block cache BinnedLRUCache@0x5637b0b41090#2 capacity: 512.00 MB usage: 1.17 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-26T04:11:55.301 INFO:journalctl@ceph.mon.a.smithi016.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.41 KB,7.7486e-05%) Misc(1,0.00 KB,0%) 2024-03-26T04:11:55.301 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-03-26T04:11:55.301 INFO:journalctl@ceph.mon.a.smithi016.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T04:11:55.301 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: starting mon.a rank 0 at public addrs v1:172.21.15.16:6789/0 at bind addrs v1:172.21.15.16:6789/0 mon_data /var/lib/ceph/mon/ceph-a fsid d7d59826-eb26-11ee-95d0-87774f69a715 2024-03-26T04:11:55.302 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: mon.a@-1(???) e1 preinit fsid d7d59826-eb26-11ee-95d0-87774f69a715 2024-03-26T04:11:55.302 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: mon.a@-1(???).mds e1 new map 2024-03-26T04:11:55.302 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: mon.a@-1(???).mds e1 print_map 2024-03-26T04:11:55.302 INFO:journalctl@ceph.mon.a.smithi016.stdout: e1 2024-03-26T04:11:55.302 INFO:journalctl@ceph.mon.a.smithi016.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T04:11:55.302 INFO:journalctl@ceph.mon.a.smithi016.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-26T04:11:55.302 INFO:journalctl@ceph.mon.a.smithi016.stdout: legacy client fscid: -1 2024-03-26T04:11:55.302 INFO:journalctl@ceph.mon.a.smithi016.stdout: 2024-03-26T04:11:55.302 INFO:journalctl@ceph.mon.a.smithi016.stdout: No filesystems configured 2024-03-26T04:11:55.302 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-26T04:11:55.302 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T04:11:55.302 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T04:11:55.303 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T04:11:55.303 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-26T04:11:55.303 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 bash[29082]: 4eb1141880f76e78f97440d7f4a860c981dc4f24726fca4ecbbccdaa2b3b8cea 2024-03-26T04:11:55.303 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 systemd[1]: Started Ceph mon.a for d7d59826-eb26-11ee-95d0-87774f69a715. 2024-03-26T04:11:55.303 INFO:teuthology.orchestra.run.smithi016.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-26T04:11:55.595 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-26T04:11:55.595 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: monmap e1: 1 mons at {a=v1:172.21.15.16:6789/0} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:11:55.595 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: fsmap 2024-03-26T04:11:55.596 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: osdmap e1: 0 total, 0 up, 0 in 2024-03-26T04:11:55.596 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:55 smithi016 ceph-mon[29101]: mgrmap e1: no daemons active 2024-03-26T04:11:56.314 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:55.698+0000 7fe552eb3700 1 Processor -- start 2024-03-26T04:11:56.315 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:55.698+0000 7fe552eb3700 1 -- start start 2024-03-26T04:11:56.315 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:55.698+0000 7fe552eb3700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe54c079ab0 con 0x7fe54c07aeb0 2024-03-26T04:11:56.315 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:55.699+0000 7fe550c4f700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe54c07aeb0 0x7fe54c079310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37780/0 (socket says 172.21.15.16:37780) 2024-03-26T04:11:56.315 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:55.699+0000 7fe550c4f700 1 -- 172.21.15.16:0/527857325 learned_addr learned my addr 172.21.15.16:0/527857325 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:11:56.315 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:55.699+0000 7fe54b7fe700 1 -- 172.21.15.16:0/527857325 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1050616190 0 0) 0x7fe54c079ab0 con 0x7fe54c07aeb0 2024-03-26T04:11:56.315 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:55.699+0000 7fe54b7fe700 1 -- 172.21.15.16:0/527857325 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5340035b0 con 0x7fe54c07aeb0 2024-03-26T04:11:56.315 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:55.699+0000 7fe54b7fe700 1 -- 172.21.15.16:0/527857325 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2926369572 0 0) 0x7fe5340035b0 con 0x7fe54c07aeb0 2024-03-26T04:11:56.315 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:55.699+0000 7fe54b7fe700 1 -- 172.21.15.16:0/527857325 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe54c079c40 con 0x7fe54c07aeb0 2024-03-26T04:11:56.315 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:55.699+0000 7fe54b7fe700 1 -- 172.21.15.16:0/527857325 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7fe544001900 con 0x7fe54c07aeb0 2024-03-26T04:11:56.315 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:55.699+0000 7fe54b7fe700 1 -- 172.21.15.16:0/527857325 <== mon.0 v1:172.21.15.16:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 848163187 0 0) 0x7fe544002060 con 0x7fe54c07aeb0 2024-03-26T04:11:56.315 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:55.699+0000 7fe54b7fe700 1 -- 172.21.15.16:0/527857325 <== mon.0 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7fe544005ce0 con 0x7fe54c07aeb0 2024-03-26T04:11:56.315 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:55.700+0000 7fe552eb3700 1 -- 172.21.15.16:0/527857325 >> v1:172.21.15.16:6789/0 conn(0x7fe54c07aeb0 legacy=0x7fe54c079310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:11:56.315 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:55.700+0000 7fe552eb3700 1 -- 172.21.15.16:0/527857325 shutdown_connections 2024-03-26T04:11:56.315 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:55.700+0000 7fe552eb3700 1 -- 172.21.15.16:0/527857325 >> 172.21.15.16:0/527857325 conn(0x7fe54c101490 msgr2=0x7fe54c103910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:11:56.315 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:55.700+0000 7fe552eb3700 1 -- 172.21.15.16:0/527857325 shutdown_connections 2024-03-26T04:11:56.316 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:55.700+0000 7fe552eb3700 1 -- 172.21.15.16:0/527857325 wait complete. 2024-03-26T04:11:56.316 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:55.701+0000 7fe552eb3700 1 Processor -- start 2024-03-26T04:11:56.316 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:55.701+0000 7fe552eb3700 1 -- start start 2024-03-26T04:11:56.316 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:55.701+0000 7fe552eb3700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe54c19c160 con 0x7fe54c07aeb0 2024-03-26T04:11:56.316 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:55.701+0000 7fe550c4f700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe54c07aeb0 0x7fe54c19ba50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37790/0 (socket says 172.21.15.16:37790) 2024-03-26T04:11:56.316 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:55.701+0000 7fe550c4f700 1 -- 172.21.15.16:0/3475536344 learned_addr learned my addr 172.21.15.16:0/3475536344 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:11:56.316 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:55.701+0000 7fe549ffb700 1 -- 172.21.15.16:0/3475536344 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1815826211 0 0) 0x7fe54c19c160 con 0x7fe54c07aeb0 2024-03-26T04:11:56.316 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:55.701+0000 7fe549ffb700 1 -- 172.21.15.16:0/3475536344 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5340055b0 con 0x7fe54c07aeb0 2024-03-26T04:11:56.316 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:55.701+0000 7fe549ffb700 1 -- 172.21.15.16:0/3475536344 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2690640394 0 0) 0x7fe5340055b0 con 0x7fe54c07aeb0 2024-03-26T04:11:56.316 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:55.701+0000 7fe549ffb700 1 -- 172.21.15.16:0/3475536344 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe54c19c160 con 0x7fe54c07aeb0 2024-03-26T04:11:56.316 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:55.701+0000 7fe549ffb700 1 -- 172.21.15.16:0/3475536344 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7fe544001400 con 0x7fe54c07aeb0 2024-03-26T04:11:56.316 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:55.701+0000 7fe549ffb700 1 -- 172.21.15.16:0/3475536344 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3786655579 0 0) 0x7fe54c19c160 con 0x7fe54c07aeb0 2024-03-26T04:11:56.316 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:55.701+0000 7fe549ffb700 1 -- 172.21.15.16:0/3475536344 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe54c19c2f0 con 0x7fe54c07aeb0 2024-03-26T04:11:56.317 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:55.702+0000 7fe552eb3700 1 -- 172.21.15.16:0/3475536344 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe54c19c5a0 con 0x7fe54c07aeb0 2024-03-26T04:11:56.317 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:55.702+0000 7fe549ffb700 1 -- 172.21.15.16:0/3475536344 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 848163187 0 0) 0x7fe5440056c0 con 0x7fe54c07aeb0 2024-03-26T04:11:56.317 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:55.702+0000 7fe549ffb700 1 -- 172.21.15.16:0/3475536344 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7fe544006530 con 0x7fe54c07aeb0 2024-03-26T04:11:56.317 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:55.702+0000 7fe549ffb700 1 -- 172.21.15.16:0/3475536344 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe544006b40 con 0x7fe54c07aeb0 2024-03-26T04:11:56.317 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:55.702+0000 7fe552eb3700 1 -- 172.21.15.16:0/3475536344 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe54c1a00f0 con 0x7fe54c07aeb0 2024-03-26T04:11:56.317 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:55.702+0000 7fe549ffb700 1 -- 172.21.15.16:0/3475536344 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 1322898288 0 0) 0x7fe5440073b0 con 0x7fe54c07aeb0 2024-03-26T04:11:56.317 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:55.703+0000 7fe552eb3700 1 -- 172.21.15.16:0/3475536344 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe54c10d3c0 con 0x7fe54c07aeb0 2024-03-26T04:11:56.317 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:55.705+0000 7fe549ffb700 1 -- 172.21.15.16:0/3475536344 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe544005f70 con 0x7fe54c07aeb0 2024-03-26T04:11:56.317 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:55.775+0000 7fe552eb3700 1 -- 172.21.15.16:0/3475536344 --> v1:172.21.15.16:6789/0 -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fe54c10d600 con 0x7fe54c07aeb0 2024-03-26T04:11:56.317 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:55.776+0000 7fe549ffb700 1 -- 172.21.15.16:0/3475536344 <== mon.0 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (unknown 534127980 0 0) 0x7fe544005aa0 con 0x7fe54c07aeb0 2024-03-26T04:11:56.317 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:55.777+0000 7fe552eb3700 1 -- 172.21.15.16:0/3475536344 >> v1:172.21.15.16:6789/0 conn(0x7fe54c07aeb0 legacy=0x7fe54c19ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:11:56.317 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:55.777+0000 7fe552eb3700 1 -- 172.21.15.16:0/3475536344 shutdown_connections 2024-03-26T04:11:56.317 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:55.777+0000 7fe552eb3700 1 -- 172.21.15.16:0/3475536344 >> 172.21.15.16:0/3475536344 conn(0x7fe54c101490 msgr2=0x7fe54c10a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:11:56.317 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:55.777+0000 7fe552eb3700 1 -- 172.21.15.16:0/3475536344 shutdown_connections 2024-03-26T04:11:56.317 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:55.777+0000 7fe552eb3700 1 -- 172.21.15.16:0/3475536344 wait complete. 2024-03-26T04:11:56.318 INFO:teuthology.orchestra.run.smithi016.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-26T04:11:56.332 INFO:teuthology.orchestra.run.smithi016.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-26T04:11:56.332 INFO:teuthology.orchestra.run.smithi016.stdout:Creating mgr... 2024-03-26T04:11:56.332 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-26T04:11:56.333 INFO:teuthology.orchestra.run.smithi016.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-26T04:11:56.577 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-d7d59826-eb26-11ee-95d0-87774f69a715@mgr.y 2024-03-26T04:11:56.577 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Failed to reset failed state of unit ceph-d7d59826-eb26-11ee-95d0-87774f69a715@mgr.y.service: Unit ceph-d7d59826-eb26-11ee-95d0-87774f69a715@mgr.y.service not loaded. 2024-03-26T04:11:56.727 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-d7d59826-eb26-11ee-95d0-87774f69a715.target.wants/ceph-d7d59826-eb26-11ee-95d0-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-d7d59826-eb26-11ee-95d0-87774f69a715@.service. 2024-03-26T04:11:56.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:56 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3475536344' entity='client.admin' 2024-03-26T04:11:57.328 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:11:57 smithi016 podman[29315]: 2024-03-26 04:11:57.16492929 +0000 UTC m=+0.319072727 container start 5d45f0a2de4bce6b21502d782b10c67257c33473df31d550c0faac81de87ad3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-26T04:11:57.328 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:11:57 smithi016 bash[29315]: 5d45f0a2de4bce6b21502d782b10c67257c33473df31d550c0faac81de87ad3e 2024-03-26T04:11:57.343 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T04:11:57.343 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout disabled 2024-03-26T04:11:57.351 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T04:11:57.351 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-03-26T04:11:57.351 INFO:teuthology.orchestra.run.smithi016.stdout:firewalld.service is not enabled 2024-03-26T04:11:57.351 INFO:teuthology.orchestra.run.smithi016.stdout:Not possible to enable service . firewalld.service is not available 2024-03-26T04:11:57.361 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T04:11:57.362 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout disabled 2024-03-26T04:11:57.370 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T04:11:57.370 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-03-26T04:11:57.370 INFO:teuthology.orchestra.run.smithi016.stdout:firewalld.service is not enabled 2024-03-26T04:11:57.370 INFO:teuthology.orchestra.run.smithi016.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-26T04:11:57.370 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mgr to start... 2024-03-26T04:11:57.370 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mgr... 2024-03-26T04:11:57.612 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:11:57 smithi016 systemd[1]: Started Ceph mgr.y for d7d59826-eb26-11ee-95d0-87774f69a715. 2024-03-26T04:11:57.866 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:11:57 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:11:57.750+0000 7f95d149c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T04:11:57.866 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:11:57 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:11:57.848+0000 7f95d149c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T04:11:58.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:11:57 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3336996982' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T04:11:58.389 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-03-26T04:11:58.389 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-03-26T04:11:58.389 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsid": "d7d59826-eb26-11ee-95d0-87774f69a715", 2024-03-26T04:11:58.389 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T04:11:58.389 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T04:11:58.389 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T04:11:58.389 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T04:11:58.389 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:11:58.389 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T04:11:58.389 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T04:11:58.390 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 0 2024-03-26T04:11:58.390 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:11:58.390 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T04:11:58.390 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T04:11:58.390 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:11:58.390 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-26T04:11:58.390 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T04:11:58.390 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:11:58.390 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T04:11:58.390 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T04:11:58.390 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:11:58.390 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T04:11:58.390 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:11:58.391 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T04:11:58.391 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T04:11:58.391 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T04:11:58.391 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T04:11:58.391 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T04:11:58.391 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T04:11:58.391 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:11:58.391 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T04:11:58.391 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T04:11:58.391 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T04:11:58.391 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T04:11:58.391 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T04:11:58.391 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T04:11:58.391 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T04:11:58.391 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T04:11:58.392 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T04:11:58.392 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:11:58.392 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T04:11:58.392 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:11:58.392 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T04:11:58.392 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T04:11:58.392 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:11:58.392 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T04:11:58.392 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-26T04:11:58.392 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T04:11:58.392 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T04:11:58.392 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T04:11:58.392 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T04:11:58.392 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T04:11:58.392 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:11:58.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T04:11:58.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:11:58.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T04:11:58.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:11:58.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T04:11:49.625516+0000", 2024-03-26T04:11:58.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T04:11:58.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:11:58.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T04:11:58.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-03-26T04:11:58.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:57.782+0000 7f1f16758700 1 Processor -- start 2024-03-26T04:11:58.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:57.782+0000 7f1f16758700 1 -- start start 2024-03-26T04:11:58.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:57.782+0000 7f1f16758700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1f10109240 con 0x7f1f10104e20 2024-03-26T04:11:58.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:57.783+0000 7f1f0ffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1f10104e20 0x7f1f10105240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37818/0 (socket says 172.21.15.16:37818) 2024-03-26T04:11:58.393 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:57.783+0000 7f1f0ffff700 1 -- 172.21.15.16:0/1831486573 learned_addr learned my addr 172.21.15.16:0/1831486573 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:11:58.394 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:57.783+0000 7f1f0effd700 1 -- 172.21.15.16:0/1831486573 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 516774822 0 0) 0x7f1f10109240 con 0x7f1f10104e20 2024-03-26T04:11:58.394 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:57.783+0000 7f1f0effd700 1 -- 172.21.15.16:0/1831486573 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1efc0035b0 con 0x7f1f10104e20 2024-03-26T04:11:58.394 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:57.783+0000 7f1f0effd700 1 -- 172.21.15.16:0/1831486573 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1877897919 0 0) 0x7f1efc0035b0 con 0x7f1f10104e20 2024-03-26T04:11:58.394 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:57.783+0000 7f1f0effd700 1 -- 172.21.15.16:0/1831486573 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f1010a3e0 con 0x7f1f10104e20 2024-03-26T04:11:58.394 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:57.783+0000 7f1f0effd700 1 -- 172.21.15.16:0/1831486573 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f1f00001580 con 0x7f1f10104e20 2024-03-26T04:11:58.394 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:57.784+0000 7f1f0effd700 1 -- 172.21.15.16:0/1831486573 <== mon.0 v1:172.21.15.16:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 848163187 0 0) 0x7f1f00001ce0 con 0x7f1f10104e20 2024-03-26T04:11:58.394 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:57.784+0000 7f1f0effd700 1 -- 172.21.15.16:0/1831486573 <== mon.0 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f1f00005960 con 0x7f1f10104e20 2024-03-26T04:11:58.394 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:57.784+0000 7f1f16758700 1 -- 172.21.15.16:0/1831486573 >> v1:172.21.15.16:6789/0 conn(0x7f1f10104e20 legacy=0x7f1f10105240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:11:58.394 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:57.784+0000 7f1f16758700 1 -- 172.21.15.16:0/1831486573 shutdown_connections 2024-03-26T04:11:58.394 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:57.784+0000 7f1f16758700 1 -- 172.21.15.16:0/1831486573 >> 172.21.15.16:0/1831486573 conn(0x7f1f101003e0 msgr2=0x7f1f10102800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:11:58.394 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:57.784+0000 7f1f16758700 1 -- 172.21.15.16:0/1831486573 shutdown_connections 2024-03-26T04:11:58.394 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:57.784+0000 7f1f16758700 1 -- 172.21.15.16:0/1831486573 wait complete. 2024-03-26T04:11:58.394 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:57.785+0000 7f1f16758700 1 Processor -- start 2024-03-26T04:11:58.395 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:57.785+0000 7f1f16758700 1 -- start start 2024-03-26T04:11:58.395 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:57.785+0000 7f1f16758700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1f1019f7b0 con 0x7f1f10104e20 2024-03-26T04:11:58.395 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:57.785+0000 7f1f0ffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1f10104e20 0x7f1f10102250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37824/0 (socket says 172.21.15.16:37824) 2024-03-26T04:11:58.395 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:57.785+0000 7f1f0ffff700 1 -- 172.21.15.16:0/3336996982 learned_addr learned my addr 172.21.15.16:0/3336996982 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:11:58.395 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:57.786+0000 7f1f0d7fa700 1 -- 172.21.15.16:0/3336996982 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1091975655 0 0) 0x7f1f1019f7b0 con 0x7f1f10104e20 2024-03-26T04:11:58.395 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:57.786+0000 7f1f0d7fa700 1 -- 172.21.15.16:0/3336996982 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1efc0055b0 con 0x7f1f10104e20 2024-03-26T04:11:58.395 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:57.786+0000 7f1f0d7fa700 1 -- 172.21.15.16:0/3336996982 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1044190257 0 0) 0x7f1efc0055b0 con 0x7f1f10104e20 2024-03-26T04:11:58.395 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:57.786+0000 7f1f0d7fa700 1 -- 172.21.15.16:0/3336996982 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f1019f7b0 con 0x7f1f10104e20 2024-03-26T04:11:58.395 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:57.786+0000 7f1f0d7fa700 1 -- 172.21.15.16:0/3336996982 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f1f00004870 con 0x7f1f10104e20 2024-03-26T04:11:58.395 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:57.786+0000 7f1f0d7fa700 1 -- 172.21.15.16:0/3336996982 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 577755210 0 0) 0x7f1f1019f7b0 con 0x7f1f10104e20 2024-03-26T04:11:58.395 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:57.786+0000 7f1f0d7fa700 1 -- 172.21.15.16:0/3336996982 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f101a0950 con 0x7f1f10104e20 2024-03-26T04:11:58.396 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:57.786+0000 7f1f16758700 1 -- 172.21.15.16:0/3336996982 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f1019f9a0 con 0x7f1f10104e20 2024-03-26T04:11:58.396 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:57.786+0000 7f1f16758700 1 -- 172.21.15.16:0/3336996982 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f1019fea0 con 0x7f1f10104e20 2024-03-26T04:11:58.396 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:57.786+0000 7f1f0d7fa700 1 -- 172.21.15.16:0/3336996982 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 848163187 0 0) 0x7f1f00004a40 con 0x7f1f10104e20 2024-03-26T04:11:58.396 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:57.786+0000 7f1f0d7fa700 1 -- 172.21.15.16:0/3336996982 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f1f00006150 con 0x7f1f10104e20 2024-03-26T04:11:58.396 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:57.786+0000 7f1f0d7fa700 1 -- 172.21.15.16:0/3336996982 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f1f00006740 con 0x7f1f10104e20 2024-03-26T04:11:58.396 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:57.786+0000 7f1f0d7fa700 1 -- 172.21.15.16:0/3336996982 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 1322898288 0 0) 0x7f1f00007e00 con 0x7f1f10104e20 2024-03-26T04:11:58.396 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:57.787+0000 7f1f16758700 1 -- 172.21.15.16:0/3336996982 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ef0005150 con 0x7f1f10104e20 2024-03-26T04:11:58.396 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:57.790+0000 7f1f0d7fa700 1 -- 172.21.15.16:0/3336996982 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f1f00005c00 con 0x7f1f10104e20 2024-03-26T04:11:58.396 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:57.864+0000 7f1f16758700 1 -- 172.21.15.16:0/3336996982 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f1ef0005400 con 0x7f1f10104e20 2024-03-26T04:11:58.396 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:57.865+0000 7f1f0d7fa700 1 -- 172.21.15.16:0/3336996982 <== mon.0 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (unknown 4201413639 0 3642131447) 0x7f1f00005640 con 0x7f1f10104e20 2024-03-26T04:11:58.396 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:57.866+0000 7f1f16758700 1 -- 172.21.15.16:0/3336996982 >> v1:172.21.15.16:6789/0 conn(0x7f1f10104e20 legacy=0x7f1f10102250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:11:58.396 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:57.866+0000 7f1f16758700 1 -- 172.21.15.16:0/3336996982 shutdown_connections 2024-03-26T04:11:58.397 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:57.866+0000 7f1f16758700 1 -- 172.21.15.16:0/3336996982 >> 172.21.15.16:0/3336996982 conn(0x7f1f101003e0 msgr2=0x7f1f10079160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:11:58.397 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:57.866+0000 7f1f16758700 1 -- 172.21.15.16:0/3336996982 shutdown_connections 2024-03-26T04:11:58.397 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:11:57.866+0000 7f1f16758700 1 -- 172.21.15.16:0/3336996982 wait complete. 2024-03-26T04:11:58.397 INFO:teuthology.orchestra.run.smithi016.stdout:mgr not available, waiting (1/15)... 2024-03-26T04:11:58.857 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:11:58 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:11:58.521+0000 7f95d149c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T04:11:58.857 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:11:58 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:11:58.616+0000 7f95d149c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T04:11:58.857 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:11:58 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:11:58.758+0000 7f95d149c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T04:11:58.857 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:11:58 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:11:58.855+0000 7f95d149c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T04:11:59.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:11:58 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:11:58.940+0000 7f95d149c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T04:11:59.414 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:11:59 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:11:59.208+0000 7f95d149c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T04:11:59.415 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:11:59 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:11:59.323+0000 7f95d149c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T04:11:59.415 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:11:59 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:11:59.413+0000 7f95d149c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T04:11:59.696 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:11:59 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:11:59.513+0000 7f95d149c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T04:12:00.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:11:59 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:11:59.695+0000 7f95d149c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T04:12:01.014 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:00 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:00.705+0000 7f95d149c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T04:12:01.014 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:00 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:00.839+0000 7f95d149c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T04:12:01.276 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:01 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2939927170' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T04:12:01.277 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:01 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:01.183+0000 7f95d149c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T04:12:01.575 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:01 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:01.275+0000 7f95d149c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T04:12:01.575 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:01 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:01.475+0000 7f95d149c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:12:01.577 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-03-26T04:12:01.580 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-03-26T04:12:01.580 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsid": "d7d59826-eb26-11ee-95d0-87774f69a715", 2024-03-26T04:12:01.580 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T04:12:01.580 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T04:12:01.581 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T04:12:01.581 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T04:12:01.581 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:12:01.581 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T04:12:01.581 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T04:12:01.581 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 0 2024-03-26T04:12:01.581 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:12:01.581 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T04:12:01.581 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T04:12:01.581 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:12:01.581 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-26T04:12:01.581 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T04:12:01.581 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:12:01.582 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T04:12:01.582 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T04:12:01.582 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:12:01.582 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T04:12:01.582 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:12:01.582 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T04:12:01.582 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T04:12:01.582 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T04:12:01.590 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T04:12:01.590 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T04:12:01.590 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T04:12:01.590 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:12:01.590 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T04:12:01.590 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T04:12:01.590 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T04:12:01.590 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T04:12:01.590 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T04:12:01.590 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T04:12:01.591 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T04:12:01.591 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T04:12:01.591 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T04:12:01.591 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:12:01.591 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T04:12:01.591 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:12:01.591 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T04:12:01.591 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T04:12:01.591 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:12:01.591 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T04:12:01.591 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-26T04:12:01.591 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T04:12:01.591 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T04:12:01.591 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T04:12:01.591 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T04:12:01.592 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T04:12:01.592 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:12:01.592 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T04:12:01.592 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:12:01.592 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T04:12:01.592 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:12:01.592 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T04:11:49.625516+0000", 2024-03-26T04:12:01.592 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T04:12:01.592 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:12:01.592 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T04:12:01.592 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-03-26T04:12:01.592 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:00.929+0000 7f1166a93700 1 Processor -- start 2024-03-26T04:12:01.592 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:00.929+0000 7f1166a93700 1 -- start start 2024-03-26T04:12:01.592 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:00.929+0000 7f1166a93700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1160079710 con 0x7f1160106a60 2024-03-26T04:12:01.593 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:00.929+0000 7f116482f700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1160106a60 0x7f1160108e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37830/0 (socket says 172.21.15.16:37830) 2024-03-26T04:12:01.593 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:00.929+0000 7f116482f700 1 -- 172.21.15.16:0/3037655431 learned_addr learned my addr 172.21.15.16:0/3037655431 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:01.593 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:00.930+0000 7f115f7fe700 1 -- 172.21.15.16:0/3037655431 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3584755279 0 0) 0x7f1160079710 con 0x7f1160106a60 2024-03-26T04:12:01.593 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:00.930+0000 7f115f7fe700 1 -- 172.21.15.16:0/3037655431 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11500035b0 con 0x7f1160106a60 2024-03-26T04:12:01.593 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:00.930+0000 7f115f7fe700 1 -- 172.21.15.16:0/3037655431 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 830050348 0 0) 0x7f11500035b0 con 0x7f1160106a60 2024-03-26T04:12:01.593 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:00.930+0000 7f115f7fe700 1 -- 172.21.15.16:0/3037655431 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f116010b580 con 0x7f1160106a60 2024-03-26T04:12:01.593 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:00.930+0000 7f115f7fe700 1 -- 172.21.15.16:0/3037655431 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f1154001580 con 0x7f1160106a60 2024-03-26T04:12:01.593 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:00.930+0000 7f115f7fe700 1 -- 172.21.15.16:0/3037655431 <== mon.0 v1:172.21.15.16:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 848163187 0 0) 0x7f1154001c10 con 0x7f1160106a60 2024-03-26T04:12:01.593 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:00.930+0000 7f115f7fe700 1 -- 172.21.15.16:0/3037655431 <== mon.0 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f11540057b0 con 0x7f1160106a60 2024-03-26T04:12:01.593 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:00.931+0000 7f1166a93700 1 -- 172.21.15.16:0/3037655431 >> v1:172.21.15.16:6789/0 conn(0x7f1160106a60 legacy=0x7f1160108e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:01.593 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:00.931+0000 7f1166a93700 1 -- 172.21.15.16:0/3037655431 shutdown_connections 2024-03-26T04:12:01.593 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:00.931+0000 7f1166a93700 1 -- 172.21.15.16:0/3037655431 >> 172.21.15.16:0/3037655431 conn(0x7f1160100490 msgr2=0x7f11601028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:01.593 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:00.931+0000 7f1166a93700 1 -- 172.21.15.16:0/3037655431 shutdown_connections 2024-03-26T04:12:01.594 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:00.931+0000 7f1166a93700 1 -- 172.21.15.16:0/3037655431 wait complete. 2024-03-26T04:12:01.594 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:00.931+0000 7f1166a93700 1 Processor -- start 2024-03-26T04:12:01.594 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:00.931+0000 7f1166a93700 1 -- start start 2024-03-26T04:12:01.594 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:00.932+0000 7f1166a93700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1160199f40 con 0x7f1160106a60 2024-03-26T04:12:01.594 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:00.932+0000 7f116482f700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1160106a60 0x7f1160199830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37844/0 (socket says 172.21.15.16:37844) 2024-03-26T04:12:01.594 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:00.932+0000 7f116482f700 1 -- 172.21.15.16:0/2939927170 learned_addr learned my addr 172.21.15.16:0/2939927170 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:01.594 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:00.932+0000 7f115dffb700 1 -- 172.21.15.16:0/2939927170 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 247811280 0 0) 0x7f1160199f40 con 0x7f1160106a60 2024-03-26T04:12:01.594 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:00.932+0000 7f115dffb700 1 -- 172.21.15.16:0/2939927170 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1150005600 con 0x7f1160106a60 2024-03-26T04:12:01.594 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:00.932+0000 7f115dffb700 1 -- 172.21.15.16:0/2939927170 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2919911235 0 0) 0x7f1150005600 con 0x7f1160106a60 2024-03-26T04:12:01.594 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:00.932+0000 7f115dffb700 1 -- 172.21.15.16:0/2939927170 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1160199f40 con 0x7f1160106a60 2024-03-26T04:12:01.594 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:00.932+0000 7f115dffb700 1 -- 172.21.15.16:0/2939927170 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f1154001080 con 0x7f1160106a60 2024-03-26T04:12:01.594 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:00.932+0000 7f115dffb700 1 -- 172.21.15.16:0/2939927170 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2387686447 0 0) 0x7f1160199f40 con 0x7f1160106a60 2024-03-26T04:12:01.594 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:00.932+0000 7f115dffb700 1 -- 172.21.15.16:0/2939927170 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f116019a0d0 con 0x7f1160106a60 2024-03-26T04:12:01.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:00.932+0000 7f1166a93700 1 -- 172.21.15.16:0/2939927170 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f116019a380 con 0x7f1160106a60 2024-03-26T04:12:01.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:00.932+0000 7f115dffb700 1 -- 172.21.15.16:0/2939927170 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 848163187 0 0) 0x7f1154004990 con 0x7f1160106a60 2024-03-26T04:12:01.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:00.932+0000 7f115dffb700 1 -- 172.21.15.16:0/2939927170 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f1154006220 con 0x7f1160106a60 2024-03-26T04:12:01.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:00.932+0000 7f115dffb700 1 -- 172.21.15.16:0/2939927170 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f1154006850 con 0x7f1160106a60 2024-03-26T04:12:01.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:00.933+0000 7f1166a93700 1 -- 172.21.15.16:0/2939927170 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f116019ded0 con 0x7f1160106a60 2024-03-26T04:12:01.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:00.933+0000 7f115dffb700 1 -- 172.21.15.16:0/2939927170 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 1322898288 0 0) 0x7f11540070c0 con 0x7f1160106a60 2024-03-26T04:12:01.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:00.933+0000 7f1166a93700 1 -- 172.21.15.16:0/2939927170 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1160193550 con 0x7f1160106a60 2024-03-26T04:12:01.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:00.935+0000 7f115dffb700 1 -- 172.21.15.16:0/2939927170 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f1154004eb0 con 0x7f1160106a60 2024-03-26T04:12:01.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:01.009+0000 7f1166a93700 1 -- 172.21.15.16:0/2939927170 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f116010b070 con 0x7f1160106a60 2024-03-26T04:12:01.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:01.010+0000 7f115dffb700 1 -- 172.21.15.16:0/2939927170 <== mon.0 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (unknown 4201413639 0 3063302473) 0x7f1154005b10 con 0x7f1160106a60 2024-03-26T04:12:01.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:01.011+0000 7f1166a93700 1 -- 172.21.15.16:0/2939927170 >> v1:172.21.15.16:6789/0 conn(0x7f1160106a60 legacy=0x7f1160199830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:01.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:01.011+0000 7f1166a93700 1 -- 172.21.15.16:0/2939927170 shutdown_connections 2024-03-26T04:12:01.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:01.011+0000 7f1166a93700 1 -- 172.21.15.16:0/2939927170 >> 172.21.15.16:0/2939927170 conn(0x7f1160100490 msgr2=0x7f1160190690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:01.595 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:01.011+0000 7f1166a93700 1 -- 172.21.15.16:0/2939927170 shutdown_connections 2024-03-26T04:12:01.596 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:01.011+0000 7f1166a93700 1 -- 172.21.15.16:0/2939927170 wait complete. 2024-03-26T04:12:01.596 INFO:teuthology.orchestra.run.smithi016.stdout:mgr not available, waiting (2/15)... 2024-03-26T04:12:01.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:01 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:01.663+0000 7f95d149c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T04:12:02.212 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:01 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:01.951+0000 7f95d149c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:12:02.212 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:02 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:02.210+0000 7f95d149c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T04:12:02.662 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:02 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:02.450+0000 7f95d149c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T04:12:03.662 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:03 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:03.275+0000 7f95d149c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T04:12:03.964 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:03 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:03.861+0000 7f95d149c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T04:12:04.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:04 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/976396702' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T04:12:04.412 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:03 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:03.962+0000 7f95d149c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T04:12:04.413 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:04 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:04.188+0000 7f95d149c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T04:12:04.728 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-03-26T04:12:04.728 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-03-26T04:12:04.728 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsid": "d7d59826-eb26-11ee-95d0-87774f69a715", 2024-03-26T04:12:04.728 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T04:12:04.728 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T04:12:04.728 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T04:12:04.729 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T04:12:04.729 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:12:04.729 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T04:12:04.729 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T04:12:04.729 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 0 2024-03-26T04:12:04.729 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:12:04.729 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T04:12:04.729 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T04:12:04.729 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:12:04.729 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-03-26T04:12:04.729 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T04:12:04.729 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:12:04.729 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T04:12:04.729 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T04:12:04.729 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:12:04.729 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T04:12:04.730 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:12:04.730 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T04:12:04.730 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T04:12:04.730 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T04:12:04.737 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T04:12:04.737 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T04:12:04.737 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T04:12:04.737 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:12:04.737 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T04:12:04.737 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T04:12:04.738 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T04:12:04.738 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T04:12:04.738 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T04:12:04.738 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T04:12:04.738 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T04:12:04.738 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T04:12:04.738 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T04:12:04.738 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:12:04.738 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T04:12:04.738 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:12:04.738 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T04:12:04.738 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T04:12:04.738 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:12:04.739 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T04:12:04.739 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-26T04:12:04.739 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T04:12:04.739 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T04:12:04.739 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T04:12:04.739 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T04:12:04.739 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T04:12:04.739 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:12:04.739 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T04:12:04.739 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:12:04.739 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T04:12:04.739 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:12:04.739 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T04:11:49.625516+0000", 2024-03-26T04:12:04.739 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T04:12:04.739 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:12:04.739 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T04:12:04.740 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-03-26T04:12:04.740 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:04.000+0000 7f74b8fa0700 1 Processor -- start 2024-03-26T04:12:04.740 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:04.000+0000 7f74b8fa0700 1 -- start start 2024-03-26T04:12:04.740 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:04.001+0000 7f74b8fa0700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f74b410a340 con 0x7f74b4105f20 2024-03-26T04:12:04.740 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:04.001+0000 7f74b259c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f74b4105f20 0x7f74b4106340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37848/0 (socket says 172.21.15.16:37848) 2024-03-26T04:12:04.740 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:04.001+0000 7f74b259c700 1 -- 172.21.15.16:0/2664116148 learned_addr learned my addr 172.21.15.16:0/2664116148 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:04.740 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:04.001+0000 7f74b159a700 1 -- 172.21.15.16:0/2664116148 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3608713304 0 0) 0x7f74b410a340 con 0x7f74b4105f20 2024-03-26T04:12:04.740 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:04.001+0000 7f74b159a700 1 -- 172.21.15.16:0/2664116148 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74a00035b0 con 0x7f74b4105f20 2024-03-26T04:12:04.740 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:04.001+0000 7f74b159a700 1 -- 172.21.15.16:0/2664116148 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1544266253 0 0) 0x7f74a00035b0 con 0x7f74b4105f20 2024-03-26T04:12:04.740 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:04.001+0000 7f74b159a700 1 -- 172.21.15.16:0/2664116148 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b410b4e0 con 0x7f74b4105f20 2024-03-26T04:12:04.740 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:04.001+0000 7f74b159a700 1 -- 172.21.15.16:0/2664116148 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f74a4001580 con 0x7f74b4105f20 2024-03-26T04:12:04.740 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:04.002+0000 7f74b159a700 1 -- 172.21.15.16:0/2664116148 <== mon.0 v1:172.21.15.16:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 848163187 0 0) 0x7f74a4001ce0 con 0x7f74b4105f20 2024-03-26T04:12:04.740 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:04.002+0000 7f74b159a700 1 -- 172.21.15.16:0/2664116148 <== mon.0 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f74a4005960 con 0x7f74b4105f20 2024-03-26T04:12:04.740 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:04.002+0000 7f74b8fa0700 1 -- 172.21.15.16:0/2664116148 >> v1:172.21.15.16:6789/0 conn(0x7f74b4105f20 legacy=0x7f74b4106340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:04.740 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:04.002+0000 7f74b8fa0700 1 -- 172.21.15.16:0/2664116148 shutdown_connections 2024-03-26T04:12:04.741 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:04.002+0000 7f74b8fa0700 1 -- 172.21.15.16:0/2664116148 >> 172.21.15.16:0/2664116148 conn(0x7f74b41014e0 msgr2=0x7f74b4103900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:04.741 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:04.002+0000 7f74b8fa0700 1 -- 172.21.15.16:0/2664116148 shutdown_connections 2024-03-26T04:12:04.741 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:04.002+0000 7f74b8fa0700 1 -- 172.21.15.16:0/2664116148 wait complete. 2024-03-26T04:12:04.741 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:04.003+0000 7f74b8fa0700 1 Processor -- start 2024-03-26T04:12:04.741 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:04.003+0000 7f74b8fa0700 1 -- start start 2024-03-26T04:12:04.741 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:04.003+0000 7f74b8fa0700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f74b419f7a0 con 0x7f74b4105f20 2024-03-26T04:12:04.741 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:04.003+0000 7f74b259c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f74b4105f20 0x7f74b407b410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37850/0 (socket says 172.21.15.16:37850) 2024-03-26T04:12:04.741 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:04.003+0000 7f74b259c700 1 -- 172.21.15.16:0/976396702 learned_addr learned my addr 172.21.15.16:0/976396702 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:04.741 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:04.003+0000 7f74ab7fe700 1 -- 172.21.15.16:0/976396702 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2855078952 0 0) 0x7f74b419f7a0 con 0x7f74b4105f20 2024-03-26T04:12:04.741 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:04.003+0000 7f74ab7fe700 1 -- 172.21.15.16:0/976396702 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74a00055b0 con 0x7f74b4105f20 2024-03-26T04:12:04.741 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:04.004+0000 7f74ab7fe700 1 -- 172.21.15.16:0/976396702 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2091433494 0 0) 0x7f74a00055b0 con 0x7f74b4105f20 2024-03-26T04:12:04.741 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:04.004+0000 7f74ab7fe700 1 -- 172.21.15.16:0/976396702 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74b419f7a0 con 0x7f74b4105f20 2024-03-26T04:12:04.741 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:04.004+0000 7f74ab7fe700 1 -- 172.21.15.16:0/976396702 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f74a4004870 con 0x7f74b4105f20 2024-03-26T04:12:04.741 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:04.004+0000 7f74ab7fe700 1 -- 172.21.15.16:0/976396702 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2448545032 0 0) 0x7f74b419f7a0 con 0x7f74b4105f20 2024-03-26T04:12:04.741 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:04.004+0000 7f74ab7fe700 1 -- 172.21.15.16:0/976396702 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b41a0940 con 0x7f74b4105f20 2024-03-26T04:12:04.742 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:04.004+0000 7f74b8fa0700 1 -- 172.21.15.16:0/976396702 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74b419f990 con 0x7f74b4105f20 2024-03-26T04:12:04.742 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:04.004+0000 7f74ab7fe700 1 -- 172.21.15.16:0/976396702 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 848163187 0 0) 0x7f74a4004a40 con 0x7f74b4105f20 2024-03-26T04:12:04.742 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:04.004+0000 7f74b8fa0700 1 -- 172.21.15.16:0/976396702 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f74b419fe90 con 0x7f74b4105f20 2024-03-26T04:12:04.742 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:04.004+0000 7f74ab7fe700 1 -- 172.21.15.16:0/976396702 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f74a4006280 con 0x7f74b4105f20 2024-03-26T04:12:04.742 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:04.004+0000 7f74ab7fe700 1 -- 172.21.15.16:0/976396702 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f74a40068e0 con 0x7f74b4105f20 2024-03-26T04:12:04.742 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:04.004+0000 7f74ab7fe700 1 -- 172.21.15.16:0/976396702 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 1322898288 0 0) 0x7f74a4007eb0 con 0x7f74b4105f20 2024-03-26T04:12:04.742 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:04.005+0000 7f74b8fa0700 1 -- 172.21.15.16:0/976396702 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74b410afd0 con 0x7f74b4105f20 2024-03-26T04:12:04.742 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:04.007+0000 7f74ab7fe700 1 -- 172.21.15.16:0/976396702 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f74a4005bf0 con 0x7f74b4105f20 2024-03-26T04:12:04.742 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:04.080+0000 7f74b8fa0700 1 -- 172.21.15.16:0/976396702 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f74b4193090 con 0x7f74b4105f20 2024-03-26T04:12:04.742 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:04.081+0000 7f74ab7fe700 1 -- 172.21.15.16:0/976396702 <== mon.0 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (unknown 4201413639 0 945345160) 0x7f74a40075a0 con 0x7f74b4105f20 2024-03-26T04:12:04.742 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:04.082+0000 7f74b8fa0700 1 -- 172.21.15.16:0/976396702 >> v1:172.21.15.16:6789/0 conn(0x7f74b4105f20 legacy=0x7f74b407b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:04.742 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:04.082+0000 7f74b8fa0700 1 -- 172.21.15.16:0/976396702 shutdown_connections 2024-03-26T04:12:04.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:04.082+0000 7f74b8fa0700 1 -- 172.21.15.16:0/976396702 >> 172.21.15.16:0/976396702 conn(0x7f74b41014e0 msgr2=0x7f74b41905e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:04.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:04.082+0000 7f74b8fa0700 1 -- 172.21.15.16:0/976396702 shutdown_connections 2024-03-26T04:12:04.743 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:04.082+0000 7f74b8fa0700 1 -- 172.21.15.16:0/976396702 wait complete. 2024-03-26T04:12:04.743 INFO:teuthology.orchestra.run.smithi016.stdout:mgr not available, waiting (3/15)... 2024-03-26T04:12:05.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:05 smithi016 ceph-mon[29101]: Activating manager daemon y 2024-03-26T04:12:05.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:05 smithi016 ceph-mon[29101]: mgrmap e2: y(active, starting, since 0.00289825s) 2024-03-26T04:12:05.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:05 smithi016 ceph-mon[29101]: from='mgr.14100 v1:172.21.15.16:0/1493910129' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T04:12:05.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:05 smithi016 ceph-mon[29101]: from='mgr.14100 v1:172.21.15.16:0/1493910129' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T04:12:05.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:05 smithi016 ceph-mon[29101]: from='mgr.14100 v1:172.21.15.16:0/1493910129' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T04:12:05.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:05 smithi016 ceph-mon[29101]: from='mgr.14100 v1:172.21.15.16:0/1493910129' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T04:12:05.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:05 smithi016 ceph-mon[29101]: from='mgr.14100 v1:172.21.15.16:0/1493910129' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T04:12:05.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:05 smithi016 ceph-mon[29101]: from='mgr.14100 v1:172.21.15.16:0/1493910129' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T04:12:05.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:05 smithi016 ceph-mon[29101]: from='mgr.14100 v1:172.21.15.16:0/1493910129' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:12:05.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:05 smithi016 ceph-mon[29101]: from='mgr.14100 v1:172.21.15.16:0/1493910129' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T04:12:05.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:05 smithi016 ceph-mon[29101]: Manager daemon y is now available 2024-03-26T04:12:05.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:05 smithi016 ceph-mon[29101]: from='mgr.14100 v1:172.21.15.16:0/1493910129' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:12:05.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:05 smithi016 ceph-mon[29101]: from='mgr.14100 v1:172.21.15.16:0/1493910129' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:12:05.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:05 smithi016 ceph-mon[29101]: from='mgr.14100 v1:172.21.15.16:0/1493910129' entity='mgr.y' 2024-03-26T04:12:05.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:05 smithi016 ceph-mon[29101]: from='mgr.14100 v1:172.21.15.16:0/1493910129' entity='mgr.y' 2024-03-26T04:12:05.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:05 smithi016 ceph-mon[29101]: from='mgr.14100 v1:172.21.15.16:0/1493910129' entity='mgr.y' 2024-03-26T04:12:06.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:06 smithi016 ceph-mon[29101]: mgrmap e3: y(active, since 1.00685s) 2024-03-26T04:12:08.061 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-03-26T04:12:08.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-03-26T04:12:08.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsid": "d7d59826-eb26-11ee-95d0-87774f69a715", 2024-03-26T04:12:08.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "health": { 2024-03-26T04:12:08.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-26T04:12:08.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-26T04:12:08.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-26T04:12:08.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:12:08.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-26T04:12:08.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-26T04:12:08.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 0 2024-03-26T04:12:08.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:12:08.062 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-26T04:12:08.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "a" 2024-03-26T04:12:08.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:12:08.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-03-26T04:12:08.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-26T04:12:08.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:12:08.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-26T04:12:08.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-26T04:12:08.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:12:08.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-26T04:12:08.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:12:08.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-26T04:12:08.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-26T04:12:08.063 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-26T04:12:08.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-26T04:12:08.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-26T04:12:08.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-26T04:12:08.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:12:08.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-26T04:12:08.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-26T04:12:08.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-26T04:12:08.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-26T04:12:08.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-26T04:12:08.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-26T04:12:08.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-26T04:12:08.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-26T04:12:08.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-26T04:12:08.064 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:12:08.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-26T04:12:08.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:12:08.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-26T04:12:08.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-26T04:12:08.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:12:08.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-26T04:12:08.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T04:12:08.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-26T04:12:08.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-26T04:12:08.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-26T04:12:08.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-26T04:12:08.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "restful" 2024-03-26T04:12:08.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ], 2024-03-26T04:12:08.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T04:12:08.065 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:12:08.066 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-26T04:12:08.066 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-26T04:12:08.066 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "modified": "2024-03-26T04:11:49.625516+0000", 2024-03-26T04:12:08.066 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-26T04:12:08.066 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout }, 2024-03-26T04:12:08.066 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-26T04:12:08.066 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-03-26T04:12:08.066 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.135+0000 7fc230089700 1 Processor -- start 2024-03-26T04:12:08.066 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.135+0000 7fc230089700 1 -- start start 2024-03-26T04:12:08.066 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.135+0000 7fc230089700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc2280796d0 con 0x7fc228104800 2024-03-26T04:12:08.066 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.135+0000 7fc22de25700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc228104800 0x7fc228104c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34078/0 (socket says 172.21.15.16:34078) 2024-03-26T04:12:08.066 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.135+0000 7fc22de25700 1 -- 172.21.15.16:0/306363818 learned_addr learned my addr 172.21.15.16:0/306363818 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:08.067 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.135+0000 7fc22ce23700 1 -- 172.21.15.16:0/306363818 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3575283247 0 0) 0x7fc2280796d0 con 0x7fc228104800 2024-03-26T04:12:08.067 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.136+0000 7fc22ce23700 1 -- 172.21.15.16:0/306363818 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2140035b0 con 0x7fc228104800 2024-03-26T04:12:08.067 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.136+0000 7fc22ce23700 1 -- 172.21.15.16:0/306363818 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 241168482 0 0) 0x7fc2140035b0 con 0x7fc228104800 2024-03-26T04:12:08.067 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.136+0000 7fc22ce23700 1 -- 172.21.15.16:0/306363818 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc228106340 con 0x7fc228104800 2024-03-26T04:12:08.067 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.136+0000 7fc22ce23700 1 -- 172.21.15.16:0/306363818 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7fc224001580 con 0x7fc228104800 2024-03-26T04:12:08.067 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.136+0000 7fc22ce23700 1 -- 172.21.15.16:0/306363818 <== mon.0 v1:172.21.15.16:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 848163187 0 0) 0x7fc224001ce0 con 0x7fc228104800 2024-03-26T04:12:08.067 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.136+0000 7fc22ce23700 1 -- 172.21.15.16:0/306363818 <== mon.0 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7fc224005960 con 0x7fc228104800 2024-03-26T04:12:08.067 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.137+0000 7fc230089700 1 -- 172.21.15.16:0/306363818 >> v1:172.21.15.16:6789/0 conn(0x7fc228104800 legacy=0x7fc228104c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:08.067 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.137+0000 7fc230089700 1 -- 172.21.15.16:0/306363818 shutdown_connections 2024-03-26T04:12:08.067 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.137+0000 7fc230089700 1 -- 172.21.15.16:0/306363818 >> 172.21.15.16:0/306363818 conn(0x7fc2281003e0 msgr2=0x7fc228102800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:08.067 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.137+0000 7fc230089700 1 -- 172.21.15.16:0/306363818 shutdown_connections 2024-03-26T04:12:08.067 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.137+0000 7fc230089700 1 -- 172.21.15.16:0/306363818 wait complete. 2024-03-26T04:12:08.067 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.137+0000 7fc230089700 1 Processor -- start 2024-03-26T04:12:08.067 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.137+0000 7fc230089700 1 -- start start 2024-03-26T04:12:08.068 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.137+0000 7fc230089700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc22819c0c0 con 0x7fc228104800 2024-03-26T04:12:08.068 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.138+0000 7fc22de25700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc228104800 0x7fc22819b9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34090/0 (socket says 172.21.15.16:34090) 2024-03-26T04:12:08.068 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.138+0000 7fc22de25700 1 -- 172.21.15.16:0/3405740136 learned_addr learned my addr 172.21.15.16:0/3405740136 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:08.068 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.138+0000 7fc21affd700 1 -- 172.21.15.16:0/3405740136 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1008485121 0 0) 0x7fc22819c0c0 con 0x7fc228104800 2024-03-26T04:12:08.068 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.138+0000 7fc21affd700 1 -- 172.21.15.16:0/3405740136 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2140055b0 con 0x7fc228104800 2024-03-26T04:12:08.068 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.138+0000 7fc21affd700 1 -- 172.21.15.16:0/3405740136 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3862239264 0 0) 0x7fc2140055b0 con 0x7fc228104800 2024-03-26T04:12:08.068 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.138+0000 7fc21affd700 1 -- 172.21.15.16:0/3405740136 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc22819c0c0 con 0x7fc228104800 2024-03-26T04:12:08.068 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.138+0000 7fc21affd700 1 -- 172.21.15.16:0/3405740136 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7fc224004870 con 0x7fc228104800 2024-03-26T04:12:08.068 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.138+0000 7fc21affd700 1 -- 172.21.15.16:0/3405740136 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3386077112 0 0) 0x7fc22819c0c0 con 0x7fc228104800 2024-03-26T04:12:08.068 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.138+0000 7fc21affd700 1 -- 172.21.15.16:0/3405740136 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc22819c250 con 0x7fc228104800 2024-03-26T04:12:08.068 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.138+0000 7fc230089700 1 -- 172.21.15.16:0/3405740136 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc22819c500 con 0x7fc228104800 2024-03-26T04:12:08.068 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.138+0000 7fc230089700 1 -- 172.21.15.16:0/3405740136 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2281a0050 con 0x7fc228104800 2024-03-26T04:12:08.068 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.138+0000 7fc21affd700 1 -- 172.21.15.16:0/3405740136 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 848163187 0 0) 0x7fc224004a40 con 0x7fc228104800 2024-03-26T04:12:08.069 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.138+0000 7fc21affd700 1 -- 172.21.15.16:0/3405740136 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7fc224006080 con 0x7fc228104800 2024-03-26T04:12:08.069 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.139+0000 7fc21affd700 1 -- 172.21.15.16:0/3405740136 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 3) v1 ==== 45776+0+0 (unknown 2495064208 0 0) 0x7fc2240114c0 con 0x7fc228104800 2024-03-26T04:12:08.069 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.139+0000 7fc21affd700 1 -- 172.21.15.16:0/3405740136 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 1322898288 0 0) 0x7fc224047d10 con 0x7fc228104800 2024-03-26T04:12:08.069 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.139+0000 7fc230089700 1 -- 172.21.15.16:0/3405740136 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc20c005150 con 0x7fc228104800 2024-03-26T04:12:08.069 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.145+0000 7fc21affd700 1 -- 172.21.15.16:0/3405740136 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fc224012cd0 con 0x7fc228104800 2024-03-26T04:12:08.069 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.208+0000 7fc21affd700 1 -- 172.21.15.16:0/3405740136 <== mon.0 v1:172.21.15.16:6789/0 10 ==== mgrmap(e 4) v1 ==== 45882+0+0 (unknown 377922316 0 0) 0x7fc224012950 con 0x7fc228104800 2024-03-26T04:12:08.069 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.436+0000 7fc230089700 1 -- 172.21.15.16:0/3405740136 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fc20c005400 con 0x7fc228104800 2024-03-26T04:12:08.069 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.437+0000 7fc21affd700 1 -- 172.21.15.16:0/3405740136 <== mon.0 v1:172.21.15.16:6789/0 11 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (unknown 4201413639 0 2328429518) 0x7fc224012ea0 con 0x7fc228104800 2024-03-26T04:12:08.069 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.440+0000 7fc230089700 1 -- 172.21.15.16:0/3405740136 >> v1:172.21.15.16:6800/3967690707 conn(0x7fc21403a100 legacy=0x7fc21403c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:08.069 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.440+0000 7fc230089700 1 -- 172.21.15.16:0/3405740136 >> v1:172.21.15.16:6789/0 conn(0x7fc228104800 legacy=0x7fc22819b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:08.069 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.440+0000 7fc230089700 1 -- 172.21.15.16:0/3405740136 shutdown_connections 2024-03-26T04:12:08.069 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.440+0000 7fc230089700 1 -- 172.21.15.16:0/3405740136 >> 172.21.15.16:0/3405740136 conn(0x7fc2281003e0 msgr2=0x7fc2281027d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:08.069 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.440+0000 7fc230089700 1 -- 172.21.15.16:0/3405740136 shutdown_connections 2024-03-26T04:12:08.069 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:07.440+0000 7fc230089700 1 -- 172.21.15.16:0/3405740136 wait complete. 2024-03-26T04:12:08.070 INFO:teuthology.orchestra.run.smithi016.stdout:mgr is available 2024-03-26T04:12:08.449 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:08 smithi016 ceph-mon[29101]: mgrmap e4: y(active, since 3s) 2024-03-26T04:12:08.449 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:08 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3405740136' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-26T04:12:09.370 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:09 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/827503963' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-26T04:12:09.377 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-03-26T04:12:09.378 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [global] 2024-03-26T04:12:09.378 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout fsid = d7d59826-eb26-11ee-95d0-87774f69a715 2024-03-26T04:12:09.378 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-26T04:12:09.378 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-26T04:12:09.378 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-26T04:12:09.378 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-26T04:12:09.378 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-03-26T04:12:09.378 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-26T04:12:09.378 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-26T04:12:09.378 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 2024-03-26T04:12:09.378 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout [osd] 2024-03-26T04:12:09.378 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-26T04:12:09.379 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-26T04:12:09.379 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:08.556+0000 7f8a76702700 1 Processor -- start 2024-03-26T04:12:09.379 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:08.556+0000 7f8a76702700 1 -- start start 2024-03-26T04:12:09.379 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:08.556+0000 7f8a76702700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8a7010a340 con 0x7f8a70105f20 2024-03-26T04:12:09.379 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:08.556+0000 7f8a6ffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8a70105f20 0x7f8a70106340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34104/0 (socket says 172.21.15.16:34104) 2024-03-26T04:12:09.379 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:08.556+0000 7f8a6ffff700 1 -- 172.21.15.16:0/562417072 learned_addr learned my addr 172.21.15.16:0/562417072 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:09.379 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:08.556+0000 7f8a6effd700 1 -- 172.21.15.16:0/562417072 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3386964572 0 0) 0x7f8a7010a340 con 0x7f8a70105f20 2024-03-26T04:12:09.379 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:08.556+0000 7f8a6effd700 1 -- 172.21.15.16:0/562417072 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a580035b0 con 0x7f8a70105f20 2024-03-26T04:12:09.379 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:08.556+0000 7f8a6effd700 1 -- 172.21.15.16:0/562417072 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3261066829 0 0) 0x7f8a580035b0 con 0x7f8a70105f20 2024-03-26T04:12:09.379 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:08.556+0000 7f8a6effd700 1 -- 172.21.15.16:0/562417072 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a7010b4e0 con 0x7f8a70105f20 2024-03-26T04:12:09.379 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:08.556+0000 7f8a6effd700 1 -- 172.21.15.16:0/562417072 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f8a60001580 con 0x7f8a70105f20 2024-03-26T04:12:09.379 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:08.557+0000 7f8a6effd700 1 -- 172.21.15.16:0/562417072 <== mon.0 v1:172.21.15.16:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 848163187 0 0) 0x7f8a60001ce0 con 0x7f8a70105f20 2024-03-26T04:12:09.379 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:08.557+0000 7f8a6effd700 1 -- 172.21.15.16:0/562417072 <== mon.0 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f8a60005960 con 0x7f8a70105f20 2024-03-26T04:12:09.379 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:08.557+0000 7f8a76702700 1 -- 172.21.15.16:0/562417072 >> v1:172.21.15.16:6789/0 conn(0x7f8a70105f20 legacy=0x7f8a70106340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:09.380 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:08.557+0000 7f8a76702700 1 -- 172.21.15.16:0/562417072 shutdown_connections 2024-03-26T04:12:09.380 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:08.557+0000 7f8a76702700 1 -- 172.21.15.16:0/562417072 >> 172.21.15.16:0/562417072 conn(0x7f8a701014e0 msgr2=0x7f8a70103900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:09.380 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:08.557+0000 7f8a76702700 1 -- 172.21.15.16:0/562417072 shutdown_connections 2024-03-26T04:12:09.380 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:08.557+0000 7f8a76702700 1 -- 172.21.15.16:0/562417072 wait complete. 2024-03-26T04:12:09.380 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:08.558+0000 7f8a76702700 1 Processor -- start 2024-03-26T04:12:09.380 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:08.558+0000 7f8a76702700 1 -- start start 2024-03-26T04:12:09.380 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:08.558+0000 7f8a76702700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8a7019f7e0 con 0x7f8a70105f20 2024-03-26T04:12:09.380 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:08.558+0000 7f8a6ffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8a70105f20 0x7f8a7007b410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34106/0 (socket says 172.21.15.16:34106) 2024-03-26T04:12:09.380 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:08.558+0000 7f8a6ffff700 1 -- 172.21.15.16:0/827503963 learned_addr learned my addr 172.21.15.16:0/827503963 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:09.380 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:08.559+0000 7f8a6d7fa700 1 -- 172.21.15.16:0/827503963 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 439951872 0 0) 0x7f8a7019f7e0 con 0x7f8a70105f20 2024-03-26T04:12:09.380 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:08.559+0000 7f8a6d7fa700 1 -- 172.21.15.16:0/827503963 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a580055b0 con 0x7f8a70105f20 2024-03-26T04:12:09.380 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:08.559+0000 7f8a6d7fa700 1 -- 172.21.15.16:0/827503963 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 169632636 0 0) 0x7f8a580055b0 con 0x7f8a70105f20 2024-03-26T04:12:09.380 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:08.559+0000 7f8a6d7fa700 1 -- 172.21.15.16:0/827503963 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a7019f7e0 con 0x7f8a70105f20 2024-03-26T04:12:09.380 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:08.559+0000 7f8a6d7fa700 1 -- 172.21.15.16:0/827503963 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f8a60004870 con 0x7f8a70105f20 2024-03-26T04:12:09.381 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:08.559+0000 7f8a6d7fa700 1 -- 172.21.15.16:0/827503963 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 746763049 0 0) 0x7f8a7019f7e0 con 0x7f8a70105f20 2024-03-26T04:12:09.381 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:08.559+0000 7f8a6d7fa700 1 -- 172.21.15.16:0/827503963 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a701a0980 con 0x7f8a70105f20 2024-03-26T04:12:09.381 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:08.559+0000 7f8a76702700 1 -- 172.21.15.16:0/827503963 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a7019f9d0 con 0x7f8a70105f20 2024-03-26T04:12:09.381 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:08.559+0000 7f8a76702700 1 -- 172.21.15.16:0/827503963 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a7019fed0 con 0x7f8a70105f20 2024-03-26T04:12:09.381 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:08.559+0000 7f8a6d7fa700 1 -- 172.21.15.16:0/827503963 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 848163187 0 0) 0x7f8a60004a40 con 0x7f8a70105f20 2024-03-26T04:12:09.381 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:08.559+0000 7f8a6d7fa700 1 -- 172.21.15.16:0/827503963 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f8a60006020 con 0x7f8a70105f20 2024-03-26T04:12:09.381 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:08.560+0000 7f8a6d7fa700 1 -- 172.21.15.16:0/827503963 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 4) v1 ==== 45882+0+0 (unknown 377922316 0 0) 0x7f8a60011510 con 0x7f8a70105f20 2024-03-26T04:12:09.381 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:08.560+0000 7f8a76702700 1 -- 172.21.15.16:0/827503963 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a5c005150 con 0x7f8a70105f20 2024-03-26T04:12:09.381 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:08.560+0000 7f8a6d7fa700 1 -- 172.21.15.16:0/827503963 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 1322898288 0 0) 0x7f8a600470a0 con 0x7f8a70105f20 2024-03-26T04:12:09.381 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:08.565+0000 7f8a6d7fa700 1 -- 172.21.15.16:0/827503963 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f8a60012d70 con 0x7f8a70105f20 2024-03-26T04:12:09.381 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:08.768+0000 7f8a76702700 1 -- 172.21.15.16:0/827503963 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f8a5c005980 con 0x7f8a70105f20 2024-03-26T04:12:09.381 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:08.769+0000 7f8a6d7fa700 1 -- 172.21.15.16:0/827503963 <== mon.0 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+282 (unknown 1187553405 0 1568053072) 0x7f8a60014e70 con 0x7f8a70105f20 2024-03-26T04:12:09.381 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:08.771+0000 7f8a76702700 1 -- 172.21.15.16:0/827503963 >> v1:172.21.15.16:6800/3967690707 conn(0x7f8a5803a1e0 legacy=0x7f8a5803c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:09.381 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:08.771+0000 7f8a76702700 1 -- 172.21.15.16:0/827503963 >> v1:172.21.15.16:6789/0 conn(0x7f8a70105f20 legacy=0x7f8a7007b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:09.382 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:08.771+0000 7f8a76702700 1 -- 172.21.15.16:0/827503963 shutdown_connections 2024-03-26T04:12:09.382 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:08.771+0000 7f8a76702700 1 -- 172.21.15.16:0/827503963 >> 172.21.15.16:0/827503963 conn(0x7f8a701014e0 msgr2=0x7f8a701038d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:09.382 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:08.771+0000 7f8a76702700 1 -- 172.21.15.16:0/827503963 shutdown_connections 2024-03-26T04:12:09.382 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:08.771+0000 7f8a76702700 1 -- 172.21.15.16:0/827503963 wait complete. 2024-03-26T04:12:09.382 INFO:teuthology.orchestra.run.smithi016.stdout:Enabling cephadm module... 2024-03-26T04:12:10.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:10 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/716079052' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-26T04:12:10.412 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:10 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ignoring --setuser ceph since I am not root 2024-03-26T04:12:10.412 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:10 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ignoring --setgroup ceph since I am not root 2024-03-26T04:12:10.825 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:09.746+0000 7f9bbc820700 1 Processor -- start 2024-03-26T04:12:10.825 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:09.746+0000 7f9bbc820700 1 -- start start 2024-03-26T04:12:10.825 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:09.746+0000 7f9bbc820700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9bb8078e60 con 0x7f9bb81069f0 2024-03-26T04:12:10.825 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:09.746+0000 7f9bb659c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9bb81069f0 0x7f9bb8108de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34116/0 (socket says 172.21.15.16:34116) 2024-03-26T04:12:10.825 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:09.746+0000 7f9bb659c700 1 -- 172.21.15.16:0/1970453470 learned_addr learned my addr 172.21.15.16:0/1970453470 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:10.825 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:09.746+0000 7f9bb559a700 1 -- 172.21.15.16:0/1970453470 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 407577813 0 0) 0x7f9bb8078e60 con 0x7f9bb81069f0 2024-03-26T04:12:10.825 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:09.746+0000 7f9bb559a700 1 -- 172.21.15.16:0/1970453470 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ba00035b0 con 0x7f9bb81069f0 2024-03-26T04:12:10.825 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:09.747+0000 7f9bb559a700 1 -- 172.21.15.16:0/1970453470 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3613620537 0 0) 0x7f9ba00035b0 con 0x7f9bb81069f0 2024-03-26T04:12:10.825 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:09.747+0000 7f9bb559a700 1 -- 172.21.15.16:0/1970453470 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb810b510 con 0x7f9bb81069f0 2024-03-26T04:12:10.825 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:09.747+0000 7f9bb559a700 1 -- 172.21.15.16:0/1970453470 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f9ba8001580 con 0x7f9bb81069f0 2024-03-26T04:12:10.825 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:09.747+0000 7f9bb559a700 1 -- 172.21.15.16:0/1970453470 <== mon.0 v1:172.21.15.16:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 848163187 0 0) 0x7f9ba8001ce0 con 0x7f9bb81069f0 2024-03-26T04:12:10.825 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:09.747+0000 7f9bb559a700 1 -- 172.21.15.16:0/1970453470 <== mon.0 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f9ba8005960 con 0x7f9bb81069f0 2024-03-26T04:12:10.826 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:09.747+0000 7f9bbc820700 1 -- 172.21.15.16:0/1970453470 >> v1:172.21.15.16:6789/0 conn(0x7f9bb81069f0 legacy=0x7f9bb8108de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:10.826 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:09.747+0000 7f9bbc820700 1 -- 172.21.15.16:0/1970453470 shutdown_connections 2024-03-26T04:12:10.826 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:09.747+0000 7f9bbc820700 1 -- 172.21.15.16:0/1970453470 >> 172.21.15.16:0/1970453470 conn(0x7f9bb81003c0 msgr2=0x7f9bb8102800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:10.826 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:09.747+0000 7f9bbc820700 1 -- 172.21.15.16:0/1970453470 shutdown_connections 2024-03-26T04:12:10.826 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:09.748+0000 7f9bbc820700 1 -- 172.21.15.16:0/1970453470 wait complete. 2024-03-26T04:12:10.826 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:09.748+0000 7f9bbc820700 1 Processor -- start 2024-03-26T04:12:10.826 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:09.748+0000 7f9bbc820700 1 -- start start 2024-03-26T04:12:10.826 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:09.748+0000 7f9bbc820700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9bb8199fa0 con 0x7f9bb81069f0 2024-03-26T04:12:10.826 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:09.749+0000 7f9bb659c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9bb81069f0 0x7f9bb8199890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34128/0 (socket says 172.21.15.16:34128) 2024-03-26T04:12:10.826 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:09.749+0000 7f9bb659c700 1 -- 172.21.15.16:0/716079052 learned_addr learned my addr 172.21.15.16:0/716079052 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:10.826 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:09.749+0000 7f9ba77fe700 1 -- 172.21.15.16:0/716079052 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2374476624 0 0) 0x7f9bb8199fa0 con 0x7f9bb81069f0 2024-03-26T04:12:10.826 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:09.749+0000 7f9ba77fe700 1 -- 172.21.15.16:0/716079052 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ba00055b0 con 0x7f9bb81069f0 2024-03-26T04:12:10.826 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:09.749+0000 7f9ba77fe700 1 -- 172.21.15.16:0/716079052 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2903141522 0 0) 0x7f9ba00055b0 con 0x7f9bb81069f0 2024-03-26T04:12:10.826 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:09.749+0000 7f9ba77fe700 1 -- 172.21.15.16:0/716079052 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9bb8199fa0 con 0x7f9bb81069f0 2024-03-26T04:12:10.827 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:09.749+0000 7f9ba77fe700 1 -- 172.21.15.16:0/716079052 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f9ba8001080 con 0x7f9bb81069f0 2024-03-26T04:12:10.827 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:09.749+0000 7f9ba77fe700 1 -- 172.21.15.16:0/716079052 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3366256515 0 0) 0x7f9bb8199fa0 con 0x7f9bb81069f0 2024-03-26T04:12:10.827 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:09.749+0000 7f9ba77fe700 1 -- 172.21.15.16:0/716079052 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb819a130 con 0x7f9bb81069f0 2024-03-26T04:12:10.827 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:09.749+0000 7f9bbc820700 1 -- 172.21.15.16:0/716079052 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bb819a3e0 con 0x7f9bb81069f0 2024-03-26T04:12:10.827 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:09.749+0000 7f9bbc820700 1 -- 172.21.15.16:0/716079052 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bb819df30 con 0x7f9bb81069f0 2024-03-26T04:12:10.827 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:09.749+0000 7f9ba77fe700 1 -- 172.21.15.16:0/716079052 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 848163187 0 0) 0x7f9ba8005340 con 0x7f9bb81069f0 2024-03-26T04:12:10.827 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:09.749+0000 7f9ba77fe700 1 -- 172.21.15.16:0/716079052 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f9ba80061b0 con 0x7f9bb81069f0 2024-03-26T04:12:10.827 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:09.750+0000 7f9ba77fe700 1 -- 172.21.15.16:0/716079052 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 4) v1 ==== 45882+0+0 (unknown 377922316 0 0) 0x7f9ba8011740 con 0x7f9bb81069f0 2024-03-26T04:12:10.827 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:09.750+0000 7f9bbc820700 1 -- 172.21.15.16:0/716079052 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bb8193530 con 0x7f9bb81069f0 2024-03-26T04:12:10.827 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:09.751+0000 7f9ba77fe700 1 -- 172.21.15.16:0/716079052 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 1322898288 0 0) 0x7f9ba80471c0 con 0x7f9bb81069f0 2024-03-26T04:12:10.827 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:09.755+0000 7f9ba77fe700 1 -- 172.21.15.16:0/716079052 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f9ba8012e20 con 0x7f9bb81069f0 2024-03-26T04:12:10.827 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:10.018+0000 7f9bbc820700 1 -- 172.21.15.16:0/716079052 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f9bb810afe0 con 0x7f9bb81069f0 2024-03-26T04:12:10.827 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:10.211+0000 7f9ba77fe700 1 -- 172.21.15.16:0/716079052 <== mon.0 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (unknown 2263024820 0 0) 0x7f9ba8014f00 con 0x7f9bb81069f0 2024-03-26T04:12:10.827 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:10.212+0000 7f9ba77fe700 1 -- 172.21.15.16:0/716079052 <== mon.0 v1:172.21.15.16:6789/0 11 ==== mgrmap(e 5) v1 ==== 45893+0+0 (unknown 1957953025 0 0) 0x7f9ba8045c10 con 0x7f9bb81069f0 2024-03-26T04:12:10.827 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:10.214+0000 7f9bbc820700 1 -- 172.21.15.16:0/716079052 >> v1:172.21.15.16:6800/3967690707 conn(0x7f9ba003a140 legacy=0x7f9ba003c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:10.828 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:10.214+0000 7f9bbc820700 1 -- 172.21.15.16:0/716079052 >> v1:172.21.15.16:6789/0 conn(0x7f9bb81069f0 legacy=0x7f9bb8199890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:10.828 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:10.214+0000 7f9bbc820700 1 -- 172.21.15.16:0/716079052 shutdown_connections 2024-03-26T04:12:10.828 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:10.214+0000 7f9bbc820700 1 -- 172.21.15.16:0/716079052 >> 172.21.15.16:0/716079052 conn(0x7f9bb81003c0 msgr2=0x7f9bb8190670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:10.828 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:10.214+0000 7f9bbc820700 1 -- 172.21.15.16:0/716079052 shutdown_connections 2024-03-26T04:12:10.828 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:10.214+0000 7f9bbc820700 1 -- 172.21.15.16:0/716079052 wait complete. 2024-03-26T04:12:11.051 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:10 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:10.791+0000 7fcf18f37200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T04:12:11.051 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:10 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:10.890+0000 7fcf18f37200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T04:12:11.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:11 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/716079052' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-26T04:12:11.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:11 smithi016 ceph-mon[29101]: mgrmap e5: y(active, since 6s) 2024-03-26T04:12:11.800 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:11 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:11.558+0000 7fcf18f37200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T04:12:11.800 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:11 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:11.654+0000 7fcf18f37200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T04:12:11.800 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:11 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:11.799+0000 7fcf18f37200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T04:12:12.054 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:11 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:11.899+0000 7fcf18f37200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T04:12:12.054 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:11 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:11.985+0000 7fcf18f37200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T04:12:12.175 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-03-26T04:12:12.175 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-26T04:12:12.175 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T04:12:12.175 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-26T04:12:12.175 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-26T04:12:12.175 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-03-26T04:12:12.175 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.261+0000 7f5ef94fa700 1 Processor -- start 2024-03-26T04:12:12.175 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.261+0000 7f5ef94fa700 1 -- start start 2024-03-26T04:12:12.175 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.261+0000 7f5ef94fa700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5ef40796d0 con 0x7f5ef4104800 2024-03-26T04:12:12.176 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.261+0000 7f5ef2ffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5ef4104800 0x7f5ef4104c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34152/0 (socket says 172.21.15.16:34152) 2024-03-26T04:12:12.176 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.261+0000 7f5ef2ffd700 1 -- 172.21.15.16:0/2065698591 learned_addr learned my addr 172.21.15.16:0/2065698591 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:12.176 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.261+0000 7f5ef1ffb700 1 -- 172.21.15.16:0/2065698591 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1236345585 0 0) 0x7f5ef40796d0 con 0x7f5ef4104800 2024-03-26T04:12:12.176 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.261+0000 7f5ef1ffb700 1 -- 172.21.15.16:0/2065698591 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5edc0035b0 con 0x7f5ef4104800 2024-03-26T04:12:12.176 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.262+0000 7f5ef1ffb700 1 -- 172.21.15.16:0/2065698591 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 50142613 0 0) 0x7f5edc0035b0 con 0x7f5ef4104800 2024-03-26T04:12:12.176 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.262+0000 7f5ef1ffb700 1 -- 172.21.15.16:0/2065698591 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef4106340 con 0x7f5ef4104800 2024-03-26T04:12:12.176 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.262+0000 7f5ef1ffb700 1 -- 172.21.15.16:0/2065698591 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f5ee4001580 con 0x7f5ef4104800 2024-03-26T04:12:12.176 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.262+0000 7f5ef1ffb700 1 -- 172.21.15.16:0/2065698591 <== mon.0 v1:172.21.15.16:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 848163187 0 0) 0x7f5ee4001ce0 con 0x7f5ef4104800 2024-03-26T04:12:12.176 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.262+0000 7f5ef1ffb700 1 -- 172.21.15.16:0/2065698591 <== mon.0 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f5ee4005960 con 0x7f5ef4104800 2024-03-26T04:12:12.176 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.262+0000 7f5ef94fa700 1 -- 172.21.15.16:0/2065698591 >> v1:172.21.15.16:6789/0 conn(0x7f5ef4104800 legacy=0x7f5ef4104c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:12.176 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.262+0000 7f5ef94fa700 1 -- 172.21.15.16:0/2065698591 shutdown_connections 2024-03-26T04:12:12.176 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.262+0000 7f5ef94fa700 1 -- 172.21.15.16:0/2065698591 >> 172.21.15.16:0/2065698591 conn(0x7f5ef41003e0 msgr2=0x7f5ef4102800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:12.176 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.263+0000 7f5ef94fa700 1 -- 172.21.15.16:0/2065698591 shutdown_connections 2024-03-26T04:12:12.176 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.263+0000 7f5ef94fa700 1 -- 172.21.15.16:0/2065698591 wait complete. 2024-03-26T04:12:12.176 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.263+0000 7f5ef94fa700 1 Processor -- start 2024-03-26T04:12:12.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.263+0000 7f5ef94fa700 1 -- start start 2024-03-26T04:12:12.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.263+0000 7f5ef94fa700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5ef419c0c0 con 0x7f5ef4104800 2024-03-26T04:12:12.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.264+0000 7f5ef2ffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5ef4104800 0x7f5ef419b9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34154/0 (socket says 172.21.15.16:34154) 2024-03-26T04:12:12.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.264+0000 7f5ef2ffd700 1 -- 172.21.15.16:0/3933989589 learned_addr learned my addr 172.21.15.16:0/3933989589 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:12.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.264+0000 7f5edbfff700 1 -- 172.21.15.16:0/3933989589 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 739336821 0 0) 0x7f5ef419c0c0 con 0x7f5ef4104800 2024-03-26T04:12:12.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.264+0000 7f5edbfff700 1 -- 172.21.15.16:0/3933989589 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5edc0055b0 con 0x7f5ef4104800 2024-03-26T04:12:12.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.264+0000 7f5edbfff700 1 -- 172.21.15.16:0/3933989589 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1084956043 0 0) 0x7f5edc0055b0 con 0x7f5ef4104800 2024-03-26T04:12:12.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.264+0000 7f5edbfff700 1 -- 172.21.15.16:0/3933989589 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ef419c0c0 con 0x7f5ef4104800 2024-03-26T04:12:12.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.264+0000 7f5edbfff700 1 -- 172.21.15.16:0/3933989589 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f5ee4004870 con 0x7f5ef4104800 2024-03-26T04:12:12.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.264+0000 7f5edbfff700 1 -- 172.21.15.16:0/3933989589 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1655761604 0 0) 0x7f5ef419c0c0 con 0x7f5ef4104800 2024-03-26T04:12:12.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.264+0000 7f5edbfff700 1 -- 172.21.15.16:0/3933989589 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef419c250 con 0x7f5ef4104800 2024-03-26T04:12:12.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.264+0000 7f5ef94fa700 1 -- 172.21.15.16:0/3933989589 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ef419c500 con 0x7f5ef4104800 2024-03-26T04:12:12.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.264+0000 7f5ef94fa700 1 -- 172.21.15.16:0/3933989589 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ef41a0050 con 0x7f5ef4104800 2024-03-26T04:12:12.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.264+0000 7f5edbfff700 1 -- 172.21.15.16:0/3933989589 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 848163187 0 0) 0x7f5ee4004a40 con 0x7f5ef4104800 2024-03-26T04:12:12.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.264+0000 7f5edbfff700 1 -- 172.21.15.16:0/3933989589 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f5ee4006060 con 0x7f5ef4104800 2024-03-26T04:12:12.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.265+0000 7f5edbfff700 1 -- 172.21.15.16:0/3933989589 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 5) v1 ==== 45893+0+0 (unknown 1957953025 0 0) 0x7f5ee40061c0 con 0x7f5ef4104800 2024-03-26T04:12:12.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.265+0000 7f5ef94fa700 1 -- 172.21.15.16:0/3933989589 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ee0005150 con 0x7f5ef4104800 2024-03-26T04:12:12.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.265+0000 7f5ef27fc700 1 -- 172.21.15.16:0/3933989589 >> v1:172.21.15.16:6800/3967690707 conn(0x7f5edc03a290 legacy=0x7f5edc03c750 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.16:6800/3967690707 2024-03-26T04:12:12.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.265+0000 7f5edbfff700 1 -- 172.21.15.16:0/3933989589 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 1322898288 0 0) 0x7f5ee4047fe0 con 0x7f5ef4104800 2024-03-26T04:12:12.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.270+0000 7f5edbfff700 1 -- 172.21.15.16:0/3933989589 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f5ee4006a20 con 0x7f5ef4104800 2024-03-26T04:12:12.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.466+0000 7f5ef27fc700 1 -- 172.21.15.16:0/3933989589 >> v1:172.21.15.16:6800/3967690707 conn(0x7f5edc03a290 legacy=0x7f5edc03c750 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.16:6800/3967690707 2024-03-26T04:12:12.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.532+0000 7f5ef94fa700 1 -- 172.21.15.16:0/3933989589 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f5ee0005c10 con 0x7f5ef4104800 2024-03-26T04:12:12.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.533+0000 7f5edbfff700 1 -- 172.21.15.16:0/3933989589 <== mon.0 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (unknown 3768197548 0 15966916) 0x7f5ee4014e50 con 0x7f5ef4104800 2024-03-26T04:12:12.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.535+0000 7f5ef94fa700 1 -- 172.21.15.16:0/3933989589 >> v1:172.21.15.16:6800/3967690707 conn(0x7f5edc03a290 legacy=0x7f5edc03c750 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T04:12:12.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.535+0000 7f5ef94fa700 1 -- 172.21.15.16:0/3933989589 >> v1:172.21.15.16:6789/0 conn(0x7f5ef4104800 legacy=0x7f5ef419b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:12.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.535+0000 7f5ef94fa700 1 -- 172.21.15.16:0/3933989589 shutdown_connections 2024-03-26T04:12:12.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.535+0000 7f5ef94fa700 1 -- 172.21.15.16:0/3933989589 >> 172.21.15.16:0/3933989589 conn(0x7f5ef41003e0 msgr2=0x7f5ef41027d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:12.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.535+0000 7f5ef94fa700 1 -- 172.21.15.16:0/3933989589 shutdown_connections 2024-03-26T04:12:12.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:11.535+0000 7f5ef94fa700 1 -- 172.21.15.16:0/3933989589 wait complete. 2024-03-26T04:12:12.179 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for the mgr to restart... 2024-03-26T04:12:12.179 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mgr epoch 5... 2024-03-26T04:12:12.310 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:12 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3933989589' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-26T04:12:12.310 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:12 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:12.257+0000 7fcf18f37200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T04:12:12.612 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:12 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:12.370+0000 7fcf18f37200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T04:12:12.612 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:12 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:12.461+0000 7fcf18f37200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T04:12:12.612 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:12 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:12.559+0000 7fcf18f37200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T04:12:12.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:12 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:12.739+0000 7fcf18f37200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T04:12:14.037 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:13 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:13.722+0000 7fcf18f37200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T04:12:14.037 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:13 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:13.855+0000 7fcf18f37200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T04:12:14.412 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:14 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:14.192+0000 7fcf18f37200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T04:12:14.412 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:14 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:14.284+0000 7fcf18f37200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T04:12:14.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:14 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:14.484+0000 7fcf18f37200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:12:14.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:14 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:14.673+0000 7fcf18f37200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T04:12:15.231 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:14 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:14.966+0000 7fcf18f37200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:12:15.231 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:15 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:15.230+0000 7fcf18f37200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T04:12:15.639 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:15 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:15.472+0000 7fcf18f37200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T04:12:16.662 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:16 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:16.296+0000 7fcf18f37200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T04:12:17.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:16 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:16.882+0000 7fcf18f37200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T04:12:17.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:16 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:16.982+0000 7fcf18f37200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T04:12:17.662 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:17 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:17.202+0000 7fcf18f37200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T04:12:17.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:17 smithi016 ceph-mon[29101]: Active manager daemon y restarted 2024-03-26T04:12:17.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:17 smithi016 ceph-mon[29101]: Activating manager daemon y 2024-03-26T04:12:17.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:17 smithi016 ceph-mon[29101]: osdmap e2: 0 total, 0 up, 0 in 2024-03-26T04:12:17.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:17 smithi016 ceph-mon[29101]: mgrmap e6: y(active, starting, since 0.00323481s) 2024-03-26T04:12:17.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:17 smithi016 ceph-mon[29101]: from='mgr.14120 v1:172.21.15.16:0/3870844279' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:12:17.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:17 smithi016 ceph-mon[29101]: from='mgr.14120 v1:172.21.15.16:0/3870844279' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T04:12:17.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:17 smithi016 ceph-mon[29101]: from='mgr.14120 v1:172.21.15.16:0/3870844279' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T04:12:17.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:17 smithi016 ceph-mon[29101]: from='mgr.14120 v1:172.21.15.16:0/3870844279' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T04:12:17.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:17 smithi016 ceph-mon[29101]: from='mgr.14120 v1:172.21.15.16:0/3870844279' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T04:12:17.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:17 smithi016 ceph-mon[29101]: Manager daemon y is now available 2024-03-26T04:12:17.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:17 smithi016 ceph-mon[29101]: Found migration_current of "None". Setting to last migration. 2024-03-26T04:12:17.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:17 smithi016 ceph-mon[29101]: from='mgr.14120 v1:172.21.15.16:0/3870844279' entity='mgr.y' 2024-03-26T04:12:17.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:17 smithi016 ceph-mon[29101]: from='mgr.14120 v1:172.21.15.16:0/3870844279' entity='mgr.y' 2024-03-26T04:12:17.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:17 smithi016 ceph-mon[29101]: from='mgr.14120 v1:172.21.15.16:0/3870844279' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:12:17.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:17 smithi016 ceph-mon[29101]: from='mgr.14120 v1:172.21.15.16:0/3870844279' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:12:17.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:17 smithi016 ceph-mon[29101]: from='mgr.14120 v1:172.21.15.16:0/3870844279' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:12:18.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:18 smithi016 ceph-mon[29101]: from='mgr.14120 v1:172.21.15.16:0/3870844279' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:12:18.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:18 smithi016 ceph-mon[29101]: from='mgr.14120 v1:172.21.15.16:0/3870844279' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:12:18.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:18 smithi016 ceph-mon[29101]: from='client.14124 v1:172.21.15.16:0/2554099173' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-26T04:12:18.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:18 smithi016 ceph-mon[29101]: mgrmap e7: y(active, since 1.0057s) 2024-03-26T04:12:18.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:18 smithi016 ceph-mon[29101]: from='client.14124 v1:172.21.15.16:0/2554099173' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-26T04:12:18.782 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-03-26T04:12:18.782 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-26T04:12:18.782 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-26T04:12:18.783 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-03-26T04:12:18.783 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:12.629+0000 7f04e6af3700 1 Processor -- start 2024-03-26T04:12:18.783 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:12.629+0000 7f04e6af3700 1 -- start start 2024-03-26T04:12:18.783 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:12.629+0000 7f04e6af3700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f04e0078e80 con 0x7f04e01069d0 2024-03-26T04:12:18.783 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:12.629+0000 7f04e488f700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f04e01069d0 0x7f04e0108dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34166/0 (socket says 172.21.15.16:34166) 2024-03-26T04:12:18.783 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:12.629+0000 7f04e488f700 1 -- 172.21.15.16:0/1357345030 learned_addr learned my addr 172.21.15.16:0/1357345030 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:18.783 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:12.630+0000 7f04df7fe700 1 -- 172.21.15.16:0/1357345030 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4168702816 0 0) 0x7f04e0078e80 con 0x7f04e01069d0 2024-03-26T04:12:18.783 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:12.630+0000 7f04df7fe700 1 -- 172.21.15.16:0/1357345030 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04d00035b0 con 0x7f04e01069d0 2024-03-26T04:12:18.783 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:12.630+0000 7f04df7fe700 1 -- 172.21.15.16:0/1357345030 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2804801864 0 0) 0x7f04d00035b0 con 0x7f04e01069d0 2024-03-26T04:12:18.783 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:12.630+0000 7f04df7fe700 1 -- 172.21.15.16:0/1357345030 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04e010b4f0 con 0x7f04e01069d0 2024-03-26T04:12:18.783 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:12.630+0000 7f04df7fe700 1 -- 172.21.15.16:0/1357345030 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f04d4001580 con 0x7f04e01069d0 2024-03-26T04:12:18.783 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:12.630+0000 7f04df7fe700 1 -- 172.21.15.16:0/1357345030 <== mon.0 v1:172.21.15.16:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 848163187 0 0) 0x7f04d4001c10 con 0x7f04e01069d0 2024-03-26T04:12:18.784 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:12.630+0000 7f04df7fe700 1 -- 172.21.15.16:0/1357345030 <== mon.0 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f04d40057b0 con 0x7f04e01069d0 2024-03-26T04:12:18.784 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:12.631+0000 7f04e6af3700 1 -- 172.21.15.16:0/1357345030 >> v1:172.21.15.16:6789/0 conn(0x7f04e01069d0 legacy=0x7f04e0108dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:18.784 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:12.631+0000 7f04e6af3700 1 -- 172.21.15.16:0/1357345030 shutdown_connections 2024-03-26T04:12:18.784 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:12.631+0000 7f04e6af3700 1 -- 172.21.15.16:0/1357345030 >> 172.21.15.16:0/1357345030 conn(0x7f04e01003c0 msgr2=0x7f04e0102800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:18.784 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:12.631+0000 7f04e6af3700 1 -- 172.21.15.16:0/1357345030 shutdown_connections 2024-03-26T04:12:18.784 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:12.631+0000 7f04e6af3700 1 -- 172.21.15.16:0/1357345030 wait complete. 2024-03-26T04:12:18.784 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:12.631+0000 7f04e6af3700 1 Processor -- start 2024-03-26T04:12:18.784 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:12.632+0000 7f04e6af3700 1 -- start start 2024-03-26T04:12:18.784 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:12.632+0000 7f04e6af3700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f04e019f8b0 con 0x7f04e01069d0 2024-03-26T04:12:18.784 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:12.632+0000 7f04e488f700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f04e01069d0 0x7f04e0106250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34168/0 (socket says 172.21.15.16:34168) 2024-03-26T04:12:18.784 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:12.632+0000 7f04e488f700 1 -- 172.21.15.16:0/2554099173 learned_addr learned my addr 172.21.15.16:0/2554099173 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:18.784 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:12.632+0000 7f04ddffb700 1 -- 172.21.15.16:0/2554099173 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1675504191 0 0) 0x7f04e019f8b0 con 0x7f04e01069d0 2024-03-26T04:12:18.784 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:12.632+0000 7f04ddffb700 1 -- 172.21.15.16:0/2554099173 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04d0005600 con 0x7f04e01069d0 2024-03-26T04:12:18.785 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:12.632+0000 7f04ddffb700 1 -- 172.21.15.16:0/2554099173 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2071444383 0 0) 0x7f04d0005600 con 0x7f04e01069d0 2024-03-26T04:12:18.785 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:12.632+0000 7f04ddffb700 1 -- 172.21.15.16:0/2554099173 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04e019f8b0 con 0x7f04e01069d0 2024-03-26T04:12:18.785 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:12.632+0000 7f04ddffb700 1 -- 172.21.15.16:0/2554099173 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f04d4001080 con 0x7f04e01069d0 2024-03-26T04:12:18.785 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:12.632+0000 7f04ddffb700 1 -- 172.21.15.16:0/2554099173 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 502217272 0 0) 0x7f04e019f8b0 con 0x7f04e01069d0 2024-03-26T04:12:18.785 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:12.632+0000 7f04ddffb700 1 -- 172.21.15.16:0/2554099173 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04e01a0a50 con 0x7f04e01069d0 2024-03-26T04:12:18.785 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:12.633+0000 7f04e6af3700 1 -- 172.21.15.16:0/2554099173 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04e019faa0 con 0x7f04e01069d0 2024-03-26T04:12:18.785 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:12.633+0000 7f04ddffb700 1 -- 172.21.15.16:0/2554099173 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 848163187 0 0) 0x7f04d4004990 con 0x7f04e01069d0 2024-03-26T04:12:18.785 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:12.633+0000 7f04ddffb700 1 -- 172.21.15.16:0/2554099173 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f04d4006220 con 0x7f04e01069d0 2024-03-26T04:12:18.785 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:12.633+0000 7f04e6af3700 1 -- 172.21.15.16:0/2554099173 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f04e019ffa0 con 0x7f04e01069d0 2024-03-26T04:12:18.785 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:12.633+0000 7f04ddffb700 1 -- 172.21.15.16:0/2554099173 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 5) v1 ==== 45893+0+0 (unknown 1957953025 0 0) 0x7f04d40117b0 con 0x7f04e01069d0 2024-03-26T04:12:18.785 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:12.634+0000 7f04e6af3700 1 -- 172.21.15.16:0/2554099173 --> v1:172.21.15.16:6800/3967690707 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04e010afe0 con 0x7f04d003a1d0 2024-03-26T04:12:18.785 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:12.634+0000 7f04ddffb700 1 -- 172.21.15.16:0/2554099173 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 1322898288 0 0) 0x7f04d4048160 con 0x7f04e01069d0 2024-03-26T04:12:18.785 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:12.634+0000 7f04dffff700 1 -- 172.21.15.16:0/2554099173 >> v1:172.21.15.16:6800/3967690707 conn(0x7f04d003a1d0 legacy=0x7f04d003c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.16:6800/3967690707 2024-03-26T04:12:18.785 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:12.835+0000 7f04dffff700 1 -- 172.21.15.16:0/2554099173 >> v1:172.21.15.16:6800/3967690707 conn(0x7f04d003a1d0 legacy=0x7f04d003c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.16:6800/3967690707 2024-03-26T04:12:18.786 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:13.236+0000 7f04dffff700 1 -- 172.21.15.16:0/2554099173 >> v1:172.21.15.16:6800/3967690707 conn(0x7f04d003a1d0 legacy=0x7f04d003c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.16:6800/3967690707 2024-03-26T04:12:18.786 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:14.037+0000 7f04dffff700 1 -- 172.21.15.16:0/2554099173 >> v1:172.21.15.16:6800/3967690707 conn(0x7f04d003a1d0 legacy=0x7f04d003c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.16:6800/3967690707 2024-03-26T04:12:18.786 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:15.638+0000 7f04dffff700 1 -- 172.21.15.16:0/2554099173 >> v1:172.21.15.16:6800/3967690707 conn(0x7f04d003a1d0 legacy=0x7f04d003c690 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.16:6800/3967690707 2024-03-26T04:12:18.786 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:17.217+0000 7f04ddffb700 1 -- 172.21.15.16:0/2554099173 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mgrmap(e 6) v1 ==== 45695+0+0 (unknown 54798757 0 0) 0x7f04d4005b10 con 0x7f04e01069d0 2024-03-26T04:12:18.786 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:17.217+0000 7f04ddffb700 1 -- 172.21.15.16:0/2554099173 >> v1:172.21.15.16:6800/3967690707 conn(0x7f04d003a1d0 legacy=0x7f04d003c690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T04:12:18.786 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:18.218+0000 7f04ddffb700 1 -- 172.21.15.16:0/2554099173 <== mon.0 v1:172.21.15.16:6789/0 10 ==== mgrmap(e 7) v1 ==== 45787+0+0 (unknown 646405796 0 0) 0x7f04d40489e0 con 0x7f04e01069d0 2024-03-26T04:12:18.786 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:18.218+0000 7f04ddffb700 1 -- 172.21.15.16:0/2554099173 --> v1:172.21.15.16:6800/1828954920 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04e010afe0 con 0x7f04d003e9f0 2024-03-26T04:12:18.786 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:18.219+0000 7f04ddffb700 1 -- 172.21.15.16:0/2554099173 <== mgr.14120 v1:172.21.15.16:6800/1828954920 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (unknown 0 0 4109177037) 0x7f04e010afe0 con 0x7f04d003e9f0 2024-03-26T04:12:18.786 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:18.226+0000 7f04e6af3700 1 -- 172.21.15.16:0/2554099173 --> v1:172.21.15.16:6800/1828954920 -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f04e010b150 con 0x7f04d003e9f0 2024-03-26T04:12:18.786 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:18.226+0000 7f04ddffb700 1 -- 172.21.15.16:0/2554099173 <== mgr.14120 v1:172.21.15.16:6800/1828954920 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (unknown 0 0 96372106) 0x7f04e010b150 con 0x7f04d003e9f0 2024-03-26T04:12:18.786 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:18.226+0000 7f04e6af3700 1 -- 172.21.15.16:0/2554099173 >> v1:172.21.15.16:6800/1828954920 conn(0x7f04d003e9f0 legacy=0x7f04d0040de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:18.786 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:18.226+0000 7f04e6af3700 1 -- 172.21.15.16:0/2554099173 >> v1:172.21.15.16:6789/0 conn(0x7f04e01069d0 legacy=0x7f04e0106250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:18.786 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:18.227+0000 7f04e6af3700 1 -- 172.21.15.16:0/2554099173 shutdown_connections 2024-03-26T04:12:18.786 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:18.227+0000 7f04e6af3700 1 -- 172.21.15.16:0/2554099173 >> 172.21.15.16:0/2554099173 conn(0x7f04e01003c0 msgr2=0x7f04e0190670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:18.787 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:18.227+0000 7f04e6af3700 1 -- 172.21.15.16:0/2554099173 shutdown_connections 2024-03-26T04:12:18.787 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:18.227+0000 7f04e6af3700 1 -- 172.21.15.16:0/2554099173 wait complete. 2024-03-26T04:12:18.787 INFO:teuthology.orchestra.run.smithi016.stdout:mgr epoch 5 is available 2024-03-26T04:12:18.787 INFO:teuthology.orchestra.run.smithi016.stdout:Setting orchestrator backend to cephadm... 2024-03-26T04:12:19.797 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:19 smithi016 ceph-mon[29101]: from='mgr.14120 v1:172.21.15.16:0/3870844279' entity='mgr.y' 2024-03-26T04:12:19.797 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:19 smithi016 ceph-mon[29101]: from='mgr.14120 v1:172.21.15.16:0/3870844279' entity='mgr.y' 2024-03-26T04:12:19.797 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:19 smithi016 ceph-mon[29101]: [26/Mar/2024:04:12:18] ENGINE Bus STARTING 2024-03-26T04:12:19.797 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:19 smithi016 ceph-mon[29101]: [26/Mar/2024:04:12:19] ENGINE Serving on http://172.21.15.16:8765 2024-03-26T04:12:19.797 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:19 smithi016 ceph-mon[29101]: [26/Mar/2024:04:12:19] ENGINE Serving on https://172.21.15.16:7150 2024-03-26T04:12:19.797 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:19 smithi016 ceph-mon[29101]: [26/Mar/2024:04:12:19] ENGINE Bus STARTED 2024-03-26T04:12:19.797 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:19 smithi016 ceph-mon[29101]: from='mgr.14120 v1:172.21.15.16:0/3870844279' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:12:19.797 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:19 smithi016 ceph-mon[29101]: from='mgr.14120 v1:172.21.15.16:0/3870844279' entity='mgr.y' 2024-03-26T04:12:19.797 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:19 smithi016 ceph-mon[29101]: from='mgr.14120 v1:172.21.15.16:0/3870844279' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:12:20.055 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:19.262+0000 7fc73004e700 1 Processor -- start 2024-03-26T04:12:20.055 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:19.262+0000 7fc73004e700 1 -- start start 2024-03-26T04:12:20.055 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:19.262+0000 7fc73004e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc72810a3d0 con 0x7fc728104210 2024-03-26T04:12:20.055 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:19.262+0000 7fc72ddea700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc728104210 0x7fc728104630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54352/0 (socket says 172.21.15.16:54352) 2024-03-26T04:12:20.055 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:19.262+0000 7fc72ddea700 1 -- 172.21.15.16:0/1154587402 learned_addr learned my addr 172.21.15.16:0/1154587402 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:20.055 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:19.262+0000 7fc72cde8700 1 -- 172.21.15.16:0/1154587402 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1730009126 0 0) 0x7fc72810a3d0 con 0x7fc728104210 2024-03-26T04:12:20.055 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:19.262+0000 7fc72cde8700 1 -- 172.21.15.16:0/1154587402 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7140035b0 con 0x7fc728104210 2024-03-26T04:12:20.055 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:19.262+0000 7fc72cde8700 1 -- 172.21.15.16:0/1154587402 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3788044880 0 0) 0x7fc7140035b0 con 0x7fc728104210 2024-03-26T04:12:20.056 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:19.263+0000 7fc72cde8700 1 -- 172.21.15.16:0/1154587402 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc72810b570 con 0x7fc728104210 2024-03-26T04:12:20.056 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:19.263+0000 7fc72cde8700 1 -- 172.21.15.16:0/1154587402 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7fc724001580 con 0x7fc728104210 2024-03-26T04:12:20.056 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:19.263+0000 7fc72cde8700 1 -- 172.21.15.16:0/1154587402 <== mon.0 v1:172.21.15.16:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 848163187 0 0) 0x7fc724001ce0 con 0x7fc728104210 2024-03-26T04:12:20.056 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:19.263+0000 7fc72cde8700 1 -- 172.21.15.16:0/1154587402 <== mon.0 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7fc724005960 con 0x7fc728104210 2024-03-26T04:12:20.056 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:19.263+0000 7fc73004e700 1 -- 172.21.15.16:0/1154587402 >> v1:172.21.15.16:6789/0 conn(0x7fc728104210 legacy=0x7fc728104630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:20.056 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:19.263+0000 7fc73004e700 1 -- 172.21.15.16:0/1154587402 shutdown_connections 2024-03-26T04:12:20.056 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:19.263+0000 7fc73004e700 1 -- 172.21.15.16:0/1154587402 >> 172.21.15.16:0/1154587402 conn(0x7fc7280ffb20 msgr2=0x7fc728101f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:20.056 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:19.263+0000 7fc73004e700 1 -- 172.21.15.16:0/1154587402 shutdown_connections 2024-03-26T04:12:20.056 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:19.263+0000 7fc73004e700 1 -- 172.21.15.16:0/1154587402 wait complete. 2024-03-26T04:12:20.056 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:19.264+0000 7fc73004e700 1 Processor -- start 2024-03-26T04:12:20.056 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:19.264+0000 7fc73004e700 1 -- start start 2024-03-26T04:12:20.056 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:19.264+0000 7fc73004e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc728199ea0 con 0x7fc728104210 2024-03-26T04:12:20.056 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:19.264+0000 7fc72ddea700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc728104210 0x7fc728199790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54360/0 (socket says 172.21.15.16:54360) 2024-03-26T04:12:20.056 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:19.264+0000 7fc72ddea700 1 -- 172.21.15.16:0/2830018894 learned_addr learned my addr 172.21.15.16:0/2830018894 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:20.057 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:19.264+0000 7fc71effd700 1 -- 172.21.15.16:0/2830018894 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 728920327 0 0) 0x7fc728199ea0 con 0x7fc728104210 2024-03-26T04:12:20.057 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:19.264+0000 7fc71effd700 1 -- 172.21.15.16:0/2830018894 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7140055b0 con 0x7fc728104210 2024-03-26T04:12:20.057 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:19.265+0000 7fc71effd700 1 -- 172.21.15.16:0/2830018894 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 173720158 0 0) 0x7fc7140055b0 con 0x7fc728104210 2024-03-26T04:12:20.057 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:19.265+0000 7fc71effd700 1 -- 172.21.15.16:0/2830018894 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc728199ea0 con 0x7fc728104210 2024-03-26T04:12:20.057 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:19.265+0000 7fc71effd700 1 -- 172.21.15.16:0/2830018894 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7fc724004870 con 0x7fc728104210 2024-03-26T04:12:20.057 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:19.265+0000 7fc71effd700 1 -- 172.21.15.16:0/2830018894 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 437961668 0 0) 0x7fc728199ea0 con 0x7fc728104210 2024-03-26T04:12:20.057 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:19.265+0000 7fc71effd700 1 -- 172.21.15.16:0/2830018894 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc72819a030 con 0x7fc728104210 2024-03-26T04:12:20.057 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:19.265+0000 7fc73004e700 1 -- 172.21.15.16:0/2830018894 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc72819a2e0 con 0x7fc728104210 2024-03-26T04:12:20.057 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:19.265+0000 7fc73004e700 1 -- 172.21.15.16:0/2830018894 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc72819de30 con 0x7fc728104210 2024-03-26T04:12:20.057 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:19.265+0000 7fc71effd700 1 -- 172.21.15.16:0/2830018894 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 848163187 0 0) 0x7fc724004a40 con 0x7fc728104210 2024-03-26T04:12:20.057 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:19.265+0000 7fc71effd700 1 -- 172.21.15.16:0/2830018894 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7fc724006030 con 0x7fc728104210 2024-03-26T04:12:20.057 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:19.266+0000 7fc71effd700 1 -- 172.21.15.16:0/2830018894 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 7) v1 ==== 45787+0+0 (unknown 646405796 0 0) 0x7fc724011480 con 0x7fc728104210 2024-03-26T04:12:20.057 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:19.266+0000 7fc73004e700 1 -- 172.21.15.16:0/2830018894 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc70c005150 con 0x7fc728104210 2024-03-26T04:12:20.057 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:19.266+0000 7fc71effd700 1 -- 172.21.15.16:0/2830018894 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (unknown 883473703 0 0) 0x7fc724046f70 con 0x7fc728104210 2024-03-26T04:12:20.058 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:19.271+0000 7fc71effd700 1 -- 172.21.15.16:0/2830018894 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fc724012ce0 con 0x7fc728104210 2024-03-26T04:12:20.058 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:19.491+0000 7fc73004e700 1 -- 172.21.15.16:0/2830018894 --> v1:172.21.15.16:6800/1828954920 -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fc70c000bc0 con 0x7fc71403a160 2024-03-26T04:12:20.058 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:19.497+0000 7fc71effd700 1 -- 172.21.15.16:0/2830018894 <== mgr.14120 v1:172.21.15.16:6800/1828954920 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (unknown 0 0 0) 0x7fc70c000bc0 con 0x7fc71403a160 2024-03-26T04:12:20.058 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:19.499+0000 7fc73004e700 1 -- 172.21.15.16:0/2830018894 >> v1:172.21.15.16:6800/1828954920 conn(0x7fc71403a160 legacy=0x7fc71403c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:20.058 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:19.499+0000 7fc73004e700 1 -- 172.21.15.16:0/2830018894 >> v1:172.21.15.16:6789/0 conn(0x7fc728104210 legacy=0x7fc728199790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:20.058 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:19.499+0000 7fc73004e700 1 -- 172.21.15.16:0/2830018894 shutdown_connections 2024-03-26T04:12:20.058 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:19.499+0000 7fc73004e700 1 -- 172.21.15.16:0/2830018894 >> 172.21.15.16:0/2830018894 conn(0x7fc7280ffb20 msgr2=0x7fc728101f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:20.058 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:19.499+0000 7fc73004e700 1 -- 172.21.15.16:0/2830018894 shutdown_connections 2024-03-26T04:12:20.058 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:19.499+0000 7fc73004e700 1 -- 172.21.15.16:0/2830018894 wait complete. 2024-03-26T04:12:20.995 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:20 smithi016 ceph-mon[29101]: from='client.14132 v1:172.21.15.16:0/2830018894' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:12:20.995 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:20 smithi016 ceph-mon[29101]: mgrmap e8: y(active, since 2s) 2024-03-26T04:12:21.269 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-26T04:12:21.269 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:20.473+0000 7f0368d8c700 1 Processor -- start 2024-03-26T04:12:21.269 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:20.473+0000 7f0368d8c700 1 -- start start 2024-03-26T04:12:21.269 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:20.473+0000 7f0368d8c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0364079750 con 0x7f0364106990 2024-03-26T04:12:21.270 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:20.473+0000 7f036259c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0364106990 0x7f0364108d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54364/0 (socket says 172.21.15.16:54364) 2024-03-26T04:12:21.270 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:20.473+0000 7f036259c700 1 -- 172.21.15.16:0/1381300407 learned_addr learned my addr 172.21.15.16:0/1381300407 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:21.270 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:20.473+0000 7f036159a700 1 -- 172.21.15.16:0/1381300407 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3225284699 0 0) 0x7f0364079750 con 0x7f0364106990 2024-03-26T04:12:21.270 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:20.474+0000 7f036159a700 1 -- 172.21.15.16:0/1381300407 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f034c0035b0 con 0x7f0364106990 2024-03-26T04:12:21.270 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:20.474+0000 7f036159a700 1 -- 172.21.15.16:0/1381300407 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1772752767 0 0) 0x7f034c0035b0 con 0x7f0364106990 2024-03-26T04:12:21.270 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:20.474+0000 7f036159a700 1 -- 172.21.15.16:0/1381300407 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f036410b4b0 con 0x7f0364106990 2024-03-26T04:12:21.270 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:20.474+0000 7f036159a700 1 -- 172.21.15.16:0/1381300407 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f0354001580 con 0x7f0364106990 2024-03-26T04:12:21.270 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:20.474+0000 7f036159a700 1 -- 172.21.15.16:0/1381300407 <== mon.0 v1:172.21.15.16:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 848163187 0 0) 0x7f0354001ce0 con 0x7f0364106990 2024-03-26T04:12:21.270 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:20.474+0000 7f036159a700 1 -- 172.21.15.16:0/1381300407 <== mon.0 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f0354005960 con 0x7f0364106990 2024-03-26T04:12:21.270 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:20.474+0000 7f0368d8c700 1 -- 172.21.15.16:0/1381300407 >> v1:172.21.15.16:6789/0 conn(0x7f0364106990 legacy=0x7f0364108d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:21.270 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:20.475+0000 7f0368d8c700 1 -- 172.21.15.16:0/1381300407 shutdown_connections 2024-03-26T04:12:21.270 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:20.475+0000 7f0368d8c700 1 -- 172.21.15.16:0/1381300407 >> 172.21.15.16:0/1381300407 conn(0x7f03641003e0 msgr2=0x7f0364102800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:21.271 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:20.475+0000 7f0368d8c700 1 -- 172.21.15.16:0/1381300407 shutdown_connections 2024-03-26T04:12:21.271 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:20.475+0000 7f0368d8c700 1 -- 172.21.15.16:0/1381300407 wait complete. 2024-03-26T04:12:21.271 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:20.475+0000 7f0368d8c700 1 Processor -- start 2024-03-26T04:12:21.271 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:20.475+0000 7f0368d8c700 1 -- start start 2024-03-26T04:12:21.271 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:20.476+0000 7f0368d8c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f0364199e60 con 0x7f0364106990 2024-03-26T04:12:21.271 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:20.476+0000 7f036259c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0364106990 0x7f0364199750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54368/0 (socket says 172.21.15.16:54368) 2024-03-26T04:12:21.271 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:20.476+0000 7f036259c700 1 -- 172.21.15.16:0/647120404 learned_addr learned my addr 172.21.15.16:0/647120404 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:21.271 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:20.476+0000 7f03537fe700 1 -- 172.21.15.16:0/647120404 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3314047584 0 0) 0x7f0364199e60 con 0x7f0364106990 2024-03-26T04:12:21.271 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:20.476+0000 7f03537fe700 1 -- 172.21.15.16:0/647120404 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f034c0055b0 con 0x7f0364106990 2024-03-26T04:12:21.271 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:20.476+0000 7f03537fe700 1 -- 172.21.15.16:0/647120404 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 703766342 0 0) 0x7f034c0055b0 con 0x7f0364106990 2024-03-26T04:12:21.271 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:20.476+0000 7f03537fe700 1 -- 172.21.15.16:0/647120404 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0364199e60 con 0x7f0364106990 2024-03-26T04:12:21.271 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:20.476+0000 7f03537fe700 1 -- 172.21.15.16:0/647120404 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f0354004870 con 0x7f0364106990 2024-03-26T04:12:21.271 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:20.476+0000 7f03537fe700 1 -- 172.21.15.16:0/647120404 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2584463752 0 0) 0x7f0364199e60 con 0x7f0364106990 2024-03-26T04:12:21.271 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:20.476+0000 7f03537fe700 1 -- 172.21.15.16:0/647120404 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0364199ff0 con 0x7f0364106990 2024-03-26T04:12:21.271 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:20.476+0000 7f0368d8c700 1 -- 172.21.15.16:0/647120404 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f036419a2a0 con 0x7f0364106990 2024-03-26T04:12:21.272 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:20.476+0000 7f0368d8c700 1 -- 172.21.15.16:0/647120404 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f036419ddf0 con 0x7f0364106990 2024-03-26T04:12:21.272 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:20.477+0000 7f03537fe700 1 -- 172.21.15.16:0/647120404 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 848163187 0 0) 0x7f0354001190 con 0x7f0364106990 2024-03-26T04:12:21.272 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:20.477+0000 7f03537fe700 1 -- 172.21.15.16:0/647120404 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f0354006120 con 0x7f0364106990 2024-03-26T04:12:21.272 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:20.477+0000 7f03537fe700 1 -- 172.21.15.16:0/647120404 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 8) v1 ==== 45893+0+0 (unknown 3089913169 0 0) 0x7f03540115f0 con 0x7f0364106990 2024-03-26T04:12:21.272 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:20.477+0000 7f0368d8c700 1 -- 172.21.15.16:0/647120404 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0344005150 con 0x7f0364106990 2024-03-26T04:12:21.272 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:20.478+0000 7f03537fe700 1 -- 172.21.15.16:0/647120404 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (unknown 883473703 0 0) 0x7f03540472b0 con 0x7f0364106990 2024-03-26T04:12:21.272 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:20.482+0000 7f03537fe700 1 -- 172.21.15.16:0/647120404 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f0354012e50 con 0x7f0364106990 2024-03-26T04:12:21.272 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:20.688+0000 7f0368d8c700 1 -- 172.21.15.16:0/647120404 --> v1:172.21.15.16:6800/1828954920 -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f0344000bc0 con 0x7f034c03a230 2024-03-26T04:12:21.272 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:20.688+0000 7f03537fe700 1 -- 172.21.15.16:0/647120404 <== mgr.14120 v1:172.21.15.16:6800/1828954920 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (unknown 0 0 2070689548) 0x7f0344000bc0 con 0x7f034c03a230 2024-03-26T04:12:21.272 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:20.690+0000 7f0368d8c700 1 -- 172.21.15.16:0/647120404 >> v1:172.21.15.16:6800/1828954920 conn(0x7f034c03a230 legacy=0x7f034c03c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:21.272 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:20.690+0000 7f0368d8c700 1 -- 172.21.15.16:0/647120404 >> v1:172.21.15.16:6789/0 conn(0x7f0364106990 legacy=0x7f0364199750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:21.272 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:20.690+0000 7f0368d8c700 1 -- 172.21.15.16:0/647120404 shutdown_connections 2024-03-26T04:12:21.272 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:20.690+0000 7f0368d8c700 1 -- 172.21.15.16:0/647120404 >> 172.21.15.16:0/647120404 conn(0x7f03641003e0 msgr2=0x7f0364190670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:21.273 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:20.691+0000 7f0368d8c700 1 -- 172.21.15.16:0/647120404 shutdown_connections 2024-03-26T04:12:21.273 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:20.691+0000 7f0368d8c700 1 -- 172.21.15.16:0/647120404 wait complete. 2024-03-26T04:12:21.273 INFO:teuthology.orchestra.run.smithi016.stdout:Generating ssh key... 2024-03-26T04:12:21.896 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:21 smithi016 ceph-mon[29101]: from='client.14134 v1:172.21.15.16:0/647120404' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:12:22.163 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:21 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: Generating public/private rsa key pair. 2024-03-26T04:12:22.163 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:21 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: Your identification has been saved in /tmp/tmpxlgl0ma0/key. 2024-03-26T04:12:22.163 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:21 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: Your public key has been saved in /tmp/tmpxlgl0ma0/key.pub. 2024-03-26T04:12:22.163 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:21 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: The key fingerprint is: 2024-03-26T04:12:22.163 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:21 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: SHA256:HFxRVIWPI+MYQnrl9lk5/HQgtOKhsviXF0Ml14eq6Dc ceph-d7d59826-eb26-11ee-95d0-87774f69a715 2024-03-26T04:12:22.163 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:21 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: The key's randomart image is: 2024-03-26T04:12:22.163 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:21 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: +---[RSA 3072]----+ 2024-03-26T04:12:22.163 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:21 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: | o=+o.+.| 2024-03-26T04:12:22.163 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:21 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: | ...o +.= .| 2024-03-26T04:12:22.163 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:21 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: | oooo+.+ * | 2024-03-26T04:12:22.164 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:21 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: | ..o+=o+ B +| 2024-03-26T04:12:22.164 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:21 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: | ..S=.* = =.| 2024-03-26T04:12:22.164 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:21 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: | . o. = + .| 2024-03-26T04:12:22.164 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:21 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: | . .. . o | 2024-03-26T04:12:22.164 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:21 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: | . + E | 2024-03-26T04:12:22.164 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:21 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: | .. o . | 2024-03-26T04:12:22.164 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:21 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: +----[SHA256]-----+ 2024-03-26T04:12:22.502 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:21.680+0000 7f67eb630700 1 Processor -- start 2024-03-26T04:12:22.502 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:21.681+0000 7f67eb630700 1 -- start start 2024-03-26T04:12:22.502 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:21.681+0000 7f67eb630700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f67e410a340 con 0x7f67e4105f20 2024-03-26T04:12:22.502 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:21.681+0000 7f67e93cc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f67e4105f20 0x7f67e4106340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54372/0 (socket says 172.21.15.16:54372) 2024-03-26T04:12:22.503 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:21.681+0000 7f67e93cc700 1 -- 172.21.15.16:0/1704971453 learned_addr learned my addr 172.21.15.16:0/1704971453 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:22.503 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:21.681+0000 7f67e3fff700 1 -- 172.21.15.16:0/1704971453 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 83077020 0 0) 0x7f67e410a340 con 0x7f67e4105f20 2024-03-26T04:12:22.503 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:21.681+0000 7f67e3fff700 1 -- 172.21.15.16:0/1704971453 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67cc0035b0 con 0x7f67e4105f20 2024-03-26T04:12:22.503 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:21.681+0000 7f67e3fff700 1 -- 172.21.15.16:0/1704971453 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2252734529 0 0) 0x7f67cc0035b0 con 0x7f67e4105f20 2024-03-26T04:12:22.503 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:21.681+0000 7f67e3fff700 1 -- 172.21.15.16:0/1704971453 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67e410b4e0 con 0x7f67e4105f20 2024-03-26T04:12:22.503 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:21.681+0000 7f67e3fff700 1 -- 172.21.15.16:0/1704971453 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f67dc001900 con 0x7f67e4105f20 2024-03-26T04:12:22.503 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:21.682+0000 7f67e3fff700 1 -- 172.21.15.16:0/1704971453 <== mon.0 v1:172.21.15.16:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 848163187 0 0) 0x7f67dc002060 con 0x7f67e4105f20 2024-03-26T04:12:22.503 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:21.682+0000 7f67e3fff700 1 -- 172.21.15.16:0/1704971453 <== mon.0 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f67dc005ce0 con 0x7f67e4105f20 2024-03-26T04:12:22.503 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:21.682+0000 7f67eb630700 1 -- 172.21.15.16:0/1704971453 >> v1:172.21.15.16:6789/0 conn(0x7f67e4105f20 legacy=0x7f67e4106340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:22.503 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:21.682+0000 7f67eb630700 1 -- 172.21.15.16:0/1704971453 shutdown_connections 2024-03-26T04:12:22.503 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:21.682+0000 7f67eb630700 1 -- 172.21.15.16:0/1704971453 >> 172.21.15.16:0/1704971453 conn(0x7f67e41014e0 msgr2=0x7f67e4103900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:22.503 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:21.682+0000 7f67eb630700 1 -- 172.21.15.16:0/1704971453 shutdown_connections 2024-03-26T04:12:22.503 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:21.682+0000 7f67eb630700 1 -- 172.21.15.16:0/1704971453 wait complete. 2024-03-26T04:12:22.504 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:21.683+0000 7f67eb630700 1 Processor -- start 2024-03-26T04:12:22.504 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:21.683+0000 7f67eb630700 1 -- start start 2024-03-26T04:12:22.504 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:21.683+0000 7f67eb630700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f67e4199ea0 con 0x7f67e4105f20 2024-03-26T04:12:22.504 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:21.683+0000 7f67e93cc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f67e4105f20 0x7f67e4199790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54384/0 (socket says 172.21.15.16:54384) 2024-03-26T04:12:22.504 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:21.683+0000 7f67e93cc700 1 -- 172.21.15.16:0/594747868 learned_addr learned my addr 172.21.15.16:0/594747868 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:22.504 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:21.683+0000 7f67e27fc700 1 -- 172.21.15.16:0/594747868 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 302765084 0 0) 0x7f67e4199ea0 con 0x7f67e4105f20 2024-03-26T04:12:22.504 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:21.683+0000 7f67e27fc700 1 -- 172.21.15.16:0/594747868 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67cc0055b0 con 0x7f67e4105f20 2024-03-26T04:12:22.504 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:21.683+0000 7f67e27fc700 1 -- 172.21.15.16:0/594747868 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3659161169 0 0) 0x7f67cc0055b0 con 0x7f67e4105f20 2024-03-26T04:12:22.504 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:21.683+0000 7f67e27fc700 1 -- 172.21.15.16:0/594747868 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67e4199ea0 con 0x7f67e4105f20 2024-03-26T04:12:22.504 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:21.683+0000 7f67e27fc700 1 -- 172.21.15.16:0/594747868 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f67dc004bf0 con 0x7f67e4105f20 2024-03-26T04:12:22.504 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:21.684+0000 7f67e27fc700 1 -- 172.21.15.16:0/594747868 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3386125542 0 0) 0x7f67e4199ea0 con 0x7f67e4105f20 2024-03-26T04:12:22.504 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:21.684+0000 7f67e27fc700 1 -- 172.21.15.16:0/594747868 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67e419a030 con 0x7f67e4105f20 2024-03-26T04:12:22.504 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:21.684+0000 7f67eb630700 1 -- 172.21.15.16:0/594747868 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67e419a2e0 con 0x7f67e4105f20 2024-03-26T04:12:22.505 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:21.684+0000 7f67eb630700 1 -- 172.21.15.16:0/594747868 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f67e419de30 con 0x7f67e4105f20 2024-03-26T04:12:22.505 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:21.684+0000 7f67e27fc700 1 -- 172.21.15.16:0/594747868 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 848163187 0 0) 0x7f67dc004dc0 con 0x7f67e4105f20 2024-03-26T04:12:22.505 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:21.684+0000 7f67e27fc700 1 -- 172.21.15.16:0/594747868 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f67dc0063f0 con 0x7f67e4105f20 2024-03-26T04:12:22.505 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:21.685+0000 7f67e27fc700 1 -- 172.21.15.16:0/594747868 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 8) v1 ==== 45893+0+0 (unknown 3089913169 0 0) 0x7f67dc0118a0 con 0x7f67e4105f20 2024-03-26T04:12:22.505 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:21.685+0000 7f67eb630700 1 -- 172.21.15.16:0/594747868 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67d0005150 con 0x7f67e4105f20 2024-03-26T04:12:22.505 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:21.685+0000 7f67e27fc700 1 -- 172.21.15.16:0/594747868 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (unknown 883473703 0 0) 0x7f67dc047530 con 0x7f67e4105f20 2024-03-26T04:12:22.505 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:21.690+0000 7f67e27fc700 1 -- 172.21.15.16:0/594747868 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f67dc013100 con 0x7f67e4105f20 2024-03-26T04:12:22.505 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:21.896+0000 7f67eb630700 1 -- 172.21.15.16:0/594747868 --> v1:172.21.15.16:6800/1828954920 -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f67d0000bc0 con 0x7f67cc03a260 2024-03-26T04:12:22.505 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:21.996+0000 7f67e27fc700 1 -- 172.21.15.16:0/594747868 <== mgr.14120 v1:172.21.15.16:6800/1828954920 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (unknown 0 0 0) 0x7f67d0000bc0 con 0x7f67cc03a260 2024-03-26T04:12:22.505 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:21.998+0000 7f67eb630700 1 -- 172.21.15.16:0/594747868 >> v1:172.21.15.16:6800/1828954920 conn(0x7f67cc03a260 legacy=0x7f67cc03c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:22.505 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:21.998+0000 7f67eb630700 1 -- 172.21.15.16:0/594747868 >> v1:172.21.15.16:6789/0 conn(0x7f67e4105f20 legacy=0x7f67e4199790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:22.505 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:21.998+0000 7f67eb630700 1 -- 172.21.15.16:0/594747868 shutdown_connections 2024-03-26T04:12:22.505 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:21.998+0000 7f67eb630700 1 -- 172.21.15.16:0/594747868 >> 172.21.15.16:0/594747868 conn(0x7f67e41014e0 msgr2=0x7f67e41038d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:22.506 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:21.998+0000 7f67eb630700 1 -- 172.21.15.16:0/594747868 shutdown_connections 2024-03-26T04:12:22.506 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:21.998+0000 7f67eb630700 1 -- 172.21.15.16:0/594747868 wait complete. 2024-03-26T04:12:23.169 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:22 smithi016 ceph-mon[29101]: from='client.14136 v1:172.21.15.16:0/594747868' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:12:23.169 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:22 smithi016 ceph-mon[29101]: Generating ssh key... 2024-03-26T04:12:23.169 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:22 smithi016 ceph-mon[29101]: from='mgr.14120 v1:172.21.15.16:0/3870844279' entity='mgr.y' 2024-03-26T04:12:23.170 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:22 smithi016 ceph-mon[29101]: from='mgr.14120 v1:172.21.15.16:0/3870844279' entity='mgr.y' 2024-03-26T04:12:23.725 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrn/eVhSc8Fpb1LKkfulVBTPWyeVKYtY2RYklcoMZHwl+T2OiaQ+STPF38Dtiv9cuOCMI2vutwsXr/T45TlOJ2+jcH20xP/9V+qAHiDQI2bypE5XA/CJPZzRVt8yuVKbXD9xQ0sVgNVTTxA8enD3Nn0CtMHjCYGehj/q3M5oTGzwIDjl6e7RX0Z2MHVBHUOZrMo9Lip90kScMeqhpNUkktfdFbLRTc0deiTSDSoJiCDhfBGayZ5TT9eFUSDogsH+1rQ6cU1bJGkJ6f/6smveDDxfxPoHXiZD4CEy/3i8cGXzmyyK/mX/swYnLclng6rvjQ5TDJ2ywP7wgyA519hSfdnnoq255MifeMK9JdXCoBJDpQkr/rTUBj3gzRm6tYidI6aK84MV6VGPSnckJ0oo1QCmp4+uEibW9nbDT6Ak573NT1QzDR/xhmXGUja3YyGyEpsLxidAhcgIbc3qMhpjLkutqM6MOt93hLYNbOcjX6puuxrGyPBMIHe2YkGmg8738= ceph-d7d59826-eb26-11ee-95d0-87774f69a715 2024-03-26T04:12:23.725 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:22.940+0000 7f088be22700 1 Processor -- start 2024-03-26T04:12:23.725 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:22.940+0000 7f088be22700 1 -- start start 2024-03-26T04:12:23.725 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:22.940+0000 7f088be22700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f088407a320 con 0x7f088407b720 2024-03-26T04:12:23.725 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:22.940+0000 7f0889bbe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f088407b720 0x7f0884079b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54396/0 (socket says 172.21.15.16:54396) 2024-03-26T04:12:23.725 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:22.940+0000 7f0889bbe700 1 -- 172.21.15.16:0/3695574753 learned_addr learned my addr 172.21.15.16:0/3695574753 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:23.725 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:22.941+0000 7f0888bbc700 1 -- 172.21.15.16:0/3695574753 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2198500354 0 0) 0x7f088407a320 con 0x7f088407b720 2024-03-26T04:12:23.725 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:22.941+0000 7f0888bbc700 1 -- 172.21.15.16:0/3695574753 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08700035b0 con 0x7f088407b720 2024-03-26T04:12:23.726 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:22.941+0000 7f0888bbc700 1 -- 172.21.15.16:0/3695574753 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2018886946 0 0) 0x7f08700035b0 con 0x7f088407b720 2024-03-26T04:12:23.726 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:22.941+0000 7f0888bbc700 1 -- 172.21.15.16:0/3695574753 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f088407a4b0 con 0x7f088407b720 2024-03-26T04:12:23.726 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:22.941+0000 7f0888bbc700 1 -- 172.21.15.16:0/3695574753 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f0880001580 con 0x7f088407b720 2024-03-26T04:12:23.726 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:22.941+0000 7f0888bbc700 1 -- 172.21.15.16:0/3695574753 <== mon.0 v1:172.21.15.16:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 848163187 0 0) 0x7f0880001ce0 con 0x7f088407b720 2024-03-26T04:12:23.726 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:22.941+0000 7f0888bbc700 1 -- 172.21.15.16:0/3695574753 <== mon.0 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f0880005960 con 0x7f088407b720 2024-03-26T04:12:23.726 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:22.941+0000 7f088be22700 1 -- 172.21.15.16:0/3695574753 >> v1:172.21.15.16:6789/0 conn(0x7f088407b720 legacy=0x7f0884079b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:23.726 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:22.942+0000 7f088be22700 1 -- 172.21.15.16:0/3695574753 shutdown_connections 2024-03-26T04:12:23.726 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:22.942+0000 7f088be22700 1 -- 172.21.15.16:0/3695574753 >> 172.21.15.16:0/3695574753 conn(0x7f08841014e0 msgr2=0x7f0884103900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:23.726 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:22.942+0000 7f088be22700 1 -- 172.21.15.16:0/3695574753 shutdown_connections 2024-03-26T04:12:23.726 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:22.942+0000 7f088be22700 1 -- 172.21.15.16:0/3695574753 wait complete. 2024-03-26T04:12:23.726 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:22.942+0000 7f088be22700 1 Processor -- start 2024-03-26T04:12:23.726 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:22.942+0000 7f088be22700 1 -- start start 2024-03-26T04:12:23.727 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:22.942+0000 7f088be22700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f088419c0c0 con 0x7f088407b720 2024-03-26T04:12:23.727 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:22.943+0000 7f0889bbe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f088407b720 0x7f088419b9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54408/0 (socket says 172.21.15.16:54408) 2024-03-26T04:12:23.727 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:22.943+0000 7f0889bbe700 1 -- 172.21.15.16:0/3537245045 learned_addr learned my addr 172.21.15.16:0/3537245045 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:23.727 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:22.943+0000 7f087affd700 1 -- 172.21.15.16:0/3537245045 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3874705428 0 0) 0x7f088419c0c0 con 0x7f088407b720 2024-03-26T04:12:23.727 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:22.943+0000 7f087affd700 1 -- 172.21.15.16:0/3537245045 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08700055b0 con 0x7f088407b720 2024-03-26T04:12:23.727 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:22.943+0000 7f087affd700 1 -- 172.21.15.16:0/3537245045 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4028869780 0 0) 0x7f08700055b0 con 0x7f088407b720 2024-03-26T04:12:23.727 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:22.943+0000 7f087affd700 1 -- 172.21.15.16:0/3537245045 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f088419c0c0 con 0x7f088407b720 2024-03-26T04:12:23.727 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:22.943+0000 7f087affd700 1 -- 172.21.15.16:0/3537245045 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f0880004870 con 0x7f088407b720 2024-03-26T04:12:23.727 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:22.943+0000 7f087affd700 1 -- 172.21.15.16:0/3537245045 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2661661539 0 0) 0x7f088419c0c0 con 0x7f088407b720 2024-03-26T04:12:23.727 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:22.943+0000 7f087affd700 1 -- 172.21.15.16:0/3537245045 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f088419c250 con 0x7f088407b720 2024-03-26T04:12:23.727 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:22.943+0000 7f088be22700 1 -- 172.21.15.16:0/3537245045 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f088419c500 con 0x7f088407b720 2024-03-26T04:12:23.727 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:22.943+0000 7f088be22700 1 -- 172.21.15.16:0/3537245045 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f08841a0050 con 0x7f088407b720 2024-03-26T04:12:23.727 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:22.943+0000 7f087affd700 1 -- 172.21.15.16:0/3537245045 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 848163187 0 0) 0x7f0880004a40 con 0x7f088407b720 2024-03-26T04:12:23.727 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:22.943+0000 7f087affd700 1 -- 172.21.15.16:0/3537245045 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f0880006290 con 0x7f088407b720 2024-03-26T04:12:23.728 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:22.944+0000 7f087affd700 1 -- 172.21.15.16:0/3537245045 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 8) v1 ==== 45893+0+0 (unknown 3089913169 0 0) 0x7f0880011760 con 0x7f088407b720 2024-03-26T04:12:23.728 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:22.944+0000 7f088be22700 1 -- 172.21.15.16:0/3537245045 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0868005150 con 0x7f088407b720 2024-03-26T04:12:23.728 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:22.944+0000 7f087affd700 1 -- 172.21.15.16:0/3537245045 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (unknown 883473703 0 0) 0x7f0880047250 con 0x7f088407b720 2024-03-26T04:12:23.728 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:22.949+0000 7f087affd700 1 -- 172.21.15.16:0/3537245045 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f0880012ee0 con 0x7f088407b720 2024-03-26T04:12:23.728 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:23.156+0000 7f088be22700 1 -- 172.21.15.16:0/3537245045 --> v1:172.21.15.16:6800/1828954920 -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f0868000bc0 con 0x7f087003a250 2024-03-26T04:12:23.728 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:23.156+0000 7f087affd700 1 -- 172.21.15.16:0/3537245045 <== mgr.14120 v1:172.21.15.16:6800/1828954920 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (unknown 0 0 1661520235) 0x7f0868000bc0 con 0x7f087003a250 2024-03-26T04:12:23.728 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:23.158+0000 7f088be22700 1 -- 172.21.15.16:0/3537245045 >> v1:172.21.15.16:6800/1828954920 conn(0x7f087003a250 legacy=0x7f087003c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:23.728 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:23.158+0000 7f088be22700 1 -- 172.21.15.16:0/3537245045 >> v1:172.21.15.16:6789/0 conn(0x7f088407b720 legacy=0x7f088419b9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:23.728 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:23.158+0000 7f088be22700 1 -- 172.21.15.16:0/3537245045 shutdown_connections 2024-03-26T04:12:23.728 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:23.158+0000 7f088be22700 1 -- 172.21.15.16:0/3537245045 >> 172.21.15.16:0/3537245045 conn(0x7f08841014e0 msgr2=0x7f088410a330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:23.728 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:23.158+0000 7f088be22700 1 -- 172.21.15.16:0/3537245045 shutdown_connections 2024-03-26T04:12:23.728 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:23.158+0000 7f088be22700 1 -- 172.21.15.16:0/3537245045 wait complete. 2024-03-26T04:12:23.728 INFO:teuthology.orchestra.run.smithi016.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-26T04:12:23.728 INFO:teuthology.orchestra.run.smithi016.stdout:Adding key to root@localhost authorized_keys... 2024-03-26T04:12:23.728 INFO:teuthology.orchestra.run.smithi016.stdout:Adding host smithi016... 2024-03-26T04:12:24.371 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:23 smithi016 ceph-mon[29101]: from='client.14138 v1:172.21.15.16:0/3537245045' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:12:25.546 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:25 smithi016 ceph-mon[29101]: from='client.14140 v1:172.21.15.16:0/2296586612' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi016", "addr": "172.21.15.16", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:12:25.546 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:25 smithi016 ceph-mon[29101]: Deploying cephadm binary to smithi016 2024-03-26T04:12:29.358 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Added host 'smithi016' with addr '172.21.15.16' 2024-03-26T04:12:29.358 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:24.150+0000 7fa86a368700 1 Processor -- start 2024-03-26T04:12:29.358 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:24.150+0000 7fa86a368700 1 -- start start 2024-03-26T04:12:29.358 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:24.150+0000 7fa86a368700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa864109260 con 0x7fa864106330 2024-03-26T04:12:29.358 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:24.151+0000 7fa863fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa864106330 0x7fa864106750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54418/0 (socket says 172.21.15.16:54418) 2024-03-26T04:12:29.358 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:24.151+0000 7fa863fff700 1 -- 172.21.15.16:0/3856696526 learned_addr learned my addr 172.21.15.16:0/3856696526 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:29.358 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:24.151+0000 7fa862ffd700 1 -- 172.21.15.16:0/3856696526 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 939180078 0 0) 0x7fa864109260 con 0x7fa864106330 2024-03-26T04:12:29.358 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:24.151+0000 7fa862ffd700 1 -- 172.21.15.16:0/3856696526 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa84c0035b0 con 0x7fa864106330 2024-03-26T04:12:29.358 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:24.151+0000 7fa862ffd700 1 -- 172.21.15.16:0/3856696526 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3926478638 0 0) 0x7fa84c0035b0 con 0x7fa864106330 2024-03-26T04:12:29.359 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:24.151+0000 7fa862ffd700 1 -- 172.21.15.16:0/3856696526 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa86410a400 con 0x7fa864106330 2024-03-26T04:12:29.359 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:24.151+0000 7fa862ffd700 1 -- 172.21.15.16:0/3856696526 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7fa854001580 con 0x7fa864106330 2024-03-26T04:12:29.359 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:24.151+0000 7fa862ffd700 1 -- 172.21.15.16:0/3856696526 <== mon.0 v1:172.21.15.16:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 848163187 0 0) 0x7fa854001ce0 con 0x7fa864106330 2024-03-26T04:12:29.359 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:24.151+0000 7fa862ffd700 1 -- 172.21.15.16:0/3856696526 <== mon.0 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7fa854005960 con 0x7fa864106330 2024-03-26T04:12:29.359 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:24.152+0000 7fa86a368700 1 -- 172.21.15.16:0/3856696526 >> v1:172.21.15.16:6789/0 conn(0x7fa864106330 legacy=0x7fa864106750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:29.359 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:24.152+0000 7fa86a368700 1 -- 172.21.15.16:0/3856696526 shutdown_connections 2024-03-26T04:12:29.359 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:24.152+0000 7fa86a368700 1 -- 172.21.15.16:0/3856696526 >> 172.21.15.16:0/3856696526 conn(0x7fa8640ffb00 msgr2=0x7fa864101f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:29.359 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:24.152+0000 7fa86a368700 1 -- 172.21.15.16:0/3856696526 shutdown_connections 2024-03-26T04:12:29.359 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:24.152+0000 7fa86a368700 1 -- 172.21.15.16:0/3856696526 wait complete. 2024-03-26T04:12:29.359 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:24.153+0000 7fa86a368700 1 Processor -- start 2024-03-26T04:12:29.359 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:24.153+0000 7fa86a368700 1 -- start start 2024-03-26T04:12:29.359 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:24.153+0000 7fa86a368700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa864199f00 con 0x7fa864106330 2024-03-26T04:12:29.359 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:24.153+0000 7fa863fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa864106330 0x7fa8641997f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54432/0 (socket says 172.21.15.16:54432) 2024-03-26T04:12:29.360 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:24.153+0000 7fa863fff700 1 -- 172.21.15.16:0/2296586612 learned_addr learned my addr 172.21.15.16:0/2296586612 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:29.360 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:24.153+0000 7fa8617fa700 1 -- 172.21.15.16:0/2296586612 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2882148140 0 0) 0x7fa864199f00 con 0x7fa864106330 2024-03-26T04:12:29.360 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:24.153+0000 7fa8617fa700 1 -- 172.21.15.16:0/2296586612 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa84c0055b0 con 0x7fa864106330 2024-03-26T04:12:29.360 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:24.154+0000 7fa8617fa700 1 -- 172.21.15.16:0/2296586612 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4270974065 0 0) 0x7fa84c0055b0 con 0x7fa864106330 2024-03-26T04:12:29.360 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:24.154+0000 7fa8617fa700 1 -- 172.21.15.16:0/2296586612 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa864199f00 con 0x7fa864106330 2024-03-26T04:12:29.360 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:24.154+0000 7fa8617fa700 1 -- 172.21.15.16:0/2296586612 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7fa854004870 con 0x7fa864106330 2024-03-26T04:12:29.360 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:24.154+0000 7fa8617fa700 1 -- 172.21.15.16:0/2296586612 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1954631461 0 0) 0x7fa864199f00 con 0x7fa864106330 2024-03-26T04:12:29.360 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:24.154+0000 7fa8617fa700 1 -- 172.21.15.16:0/2296586612 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa86419a090 con 0x7fa864106330 2024-03-26T04:12:29.360 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:24.154+0000 7fa86a368700 1 -- 172.21.15.16:0/2296586612 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa86419a340 con 0x7fa864106330 2024-03-26T04:12:29.360 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:24.154+0000 7fa86a368700 1 -- 172.21.15.16:0/2296586612 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa86419de90 con 0x7fa864106330 2024-03-26T04:12:29.360 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:24.154+0000 7fa8617fa700 1 -- 172.21.15.16:0/2296586612 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 848163187 0 0) 0x7fa854004a40 con 0x7fa864106330 2024-03-26T04:12:29.360 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:24.154+0000 7fa8617fa700 1 -- 172.21.15.16:0/2296586612 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7fa854006060 con 0x7fa864106330 2024-03-26T04:12:29.360 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:24.155+0000 7fa8617fa700 1 -- 172.21.15.16:0/2296586612 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 8) v1 ==== 45893+0+0 (unknown 3089913169 0 0) 0x7fa854011510 con 0x7fa864106330 2024-03-26T04:12:29.360 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:24.155+0000 7fa86a368700 1 -- 172.21.15.16:0/2296586612 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa850005150 con 0x7fa864106330 2024-03-26T04:12:29.361 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:24.155+0000 7fa8617fa700 1 -- 172.21.15.16:0/2296586612 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (unknown 883473703 0 0) 0x7fa8540471c0 con 0x7fa864106330 2024-03-26T04:12:29.361 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:24.160+0000 7fa8617fa700 1 -- 172.21.15.16:0/2296586612 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fa854012d30 con 0x7fa864106330 2024-03-26T04:12:29.361 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:24.370+0000 7fa86a368700 1 -- 172.21.15.16:0/2296586612 --> v1:172.21.15.16:6800/1828954920 -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi016", "addr": "172.21.15.16", "target": ["mon-mgr", ""]}) v1 -- 0x7fa850000bc0 con 0x7fa84c03a280 2024-03-26T04:12:29.361 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:28.731+0000 7fa8617fa700 1 -- 172.21.15.16:0/2296586612 <== mgr.14120 v1:172.21.15.16:6800/1828954920 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (unknown 0 0 2332247044) 0x7fa850000bc0 con 0x7fa84c03a280 2024-03-26T04:12:29.361 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:28.734+0000 7fa86a368700 1 -- 172.21.15.16:0/2296586612 >> v1:172.21.15.16:6800/1828954920 conn(0x7fa84c03a280 legacy=0x7fa84c03c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:29.361 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:28.734+0000 7fa86a368700 1 -- 172.21.15.16:0/2296586612 >> v1:172.21.15.16:6789/0 conn(0x7fa864106330 legacy=0x7fa8641997f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:29.361 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:28.734+0000 7fa86a368700 1 -- 172.21.15.16:0/2296586612 shutdown_connections 2024-03-26T04:12:29.361 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:28.734+0000 7fa86a368700 1 -- 172.21.15.16:0/2296586612 >> 172.21.15.16:0/2296586612 conn(0x7fa8640ffb00 msgr2=0x7fa864103ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:29.361 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:28.734+0000 7fa86a368700 1 -- 172.21.15.16:0/2296586612 shutdown_connections 2024-03-26T04:12:29.361 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:28.734+0000 7fa86a368700 1 -- 172.21.15.16:0/2296586612 wait complete. 2024-03-26T04:12:29.361 INFO:teuthology.orchestra.run.smithi016.stdout:Deploying unmanaged mon service... 2024-03-26T04:12:29.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:29 smithi016 ceph-mon[29101]: from='mgr.14120 v1:172.21.15.16:0/3870844279' entity='mgr.y' 2024-03-26T04:12:29.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:29 smithi016 ceph-mon[29101]: Added host smithi016 2024-03-26T04:12:29.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:29 smithi016 ceph-mon[29101]: from='mgr.14120 v1:172.21.15.16:0/3870844279' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:12:30.824 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-26T04:12:30.825 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:29.814+0000 7f4176617700 1 Processor -- start 2024-03-26T04:12:30.825 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:29.814+0000 7f4176617700 1 -- start start 2024-03-26T04:12:30.825 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:29.814+0000 7f4176617700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f41700796d0 con 0x7f41701069f0 2024-03-26T04:12:30.825 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:29.814+0000 7f416ffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f41701069f0 0x7f4170108de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54296/0 (socket says 172.21.15.16:54296) 2024-03-26T04:12:30.825 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:29.814+0000 7f416ffff700 1 -- 172.21.15.16:0/3049166604 learned_addr learned my addr 172.21.15.16:0/3049166604 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:30.825 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:29.815+0000 7f416effd700 1 -- 172.21.15.16:0/3049166604 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3628946908 0 0) 0x7f41700796d0 con 0x7f41701069f0 2024-03-26T04:12:30.825 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:29.815+0000 7f416effd700 1 -- 172.21.15.16:0/3049166604 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f415c0035b0 con 0x7f41701069f0 2024-03-26T04:12:30.825 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:29.815+0000 7f416effd700 1 -- 172.21.15.16:0/3049166604 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1324056741 0 0) 0x7f415c0035b0 con 0x7f41701069f0 2024-03-26T04:12:30.825 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:29.815+0000 7f416effd700 1 -- 172.21.15.16:0/3049166604 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f417010b510 con 0x7f41701069f0 2024-03-26T04:12:30.825 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:29.815+0000 7f416effd700 1 -- 172.21.15.16:0/3049166604 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f4160001580 con 0x7f41701069f0 2024-03-26T04:12:30.825 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:29.815+0000 7f416effd700 1 -- 172.21.15.16:0/3049166604 <== mon.0 v1:172.21.15.16:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 848163187 0 0) 0x7f4160001c10 con 0x7f41701069f0 2024-03-26T04:12:30.826 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:29.815+0000 7f416effd700 1 -- 172.21.15.16:0/3049166604 <== mon.0 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f41600057b0 con 0x7f41701069f0 2024-03-26T04:12:30.826 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:29.816+0000 7f4176617700 1 -- 172.21.15.16:0/3049166604 >> v1:172.21.15.16:6789/0 conn(0x7f41701069f0 legacy=0x7f4170108de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:30.826 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:29.816+0000 7f4176617700 1 -- 172.21.15.16:0/3049166604 shutdown_connections 2024-03-26T04:12:30.826 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:29.816+0000 7f4176617700 1 -- 172.21.15.16:0/3049166604 >> 172.21.15.16:0/3049166604 conn(0x7f4170100360 msgr2=0x7f41701027e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:30.826 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:29.816+0000 7f4176617700 1 -- 172.21.15.16:0/3049166604 shutdown_connections 2024-03-26T04:12:30.826 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:29.816+0000 7f4176617700 1 -- 172.21.15.16:0/3049166604 wait complete. 2024-03-26T04:12:30.826 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:29.816+0000 7f4176617700 1 Processor -- start 2024-03-26T04:12:30.826 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:29.816+0000 7f4176617700 1 -- start start 2024-03-26T04:12:30.826 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:29.817+0000 7f4176617700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4170199f20 con 0x7f41701069f0 2024-03-26T04:12:30.826 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:29.817+0000 7f416ffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f41701069f0 0x7f4170199810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54306/0 (socket says 172.21.15.16:54306) 2024-03-26T04:12:30.826 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:29.817+0000 7f416ffff700 1 -- 172.21.15.16:0/629545808 learned_addr learned my addr 172.21.15.16:0/629545808 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:30.826 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:29.817+0000 7f416d7fa700 1 -- 172.21.15.16:0/629545808 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2165299961 0 0) 0x7f4170199f20 con 0x7f41701069f0 2024-03-26T04:12:30.826 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:29.817+0000 7f416d7fa700 1 -- 172.21.15.16:0/629545808 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f415c005600 con 0x7f41701069f0 2024-03-26T04:12:30.827 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:29.817+0000 7f416d7fa700 1 -- 172.21.15.16:0/629545808 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3079492096 0 0) 0x7f415c005600 con 0x7f41701069f0 2024-03-26T04:12:30.827 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:29.817+0000 7f416d7fa700 1 -- 172.21.15.16:0/629545808 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4170199f20 con 0x7f41701069f0 2024-03-26T04:12:30.827 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:29.817+0000 7f416d7fa700 1 -- 172.21.15.16:0/629545808 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f4160001080 con 0x7f41701069f0 2024-03-26T04:12:30.827 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:29.817+0000 7f416d7fa700 1 -- 172.21.15.16:0/629545808 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3680426565 0 0) 0x7f4170199f20 con 0x7f41701069f0 2024-03-26T04:12:30.827 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:29.817+0000 7f416d7fa700 1 -- 172.21.15.16:0/629545808 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f417019a0b0 con 0x7f41701069f0 2024-03-26T04:12:30.827 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:29.817+0000 7f4176617700 1 -- 172.21.15.16:0/629545808 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f417019a360 con 0x7f41701069f0 2024-03-26T04:12:30.827 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:29.818+0000 7f4176617700 1 -- 172.21.15.16:0/629545808 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f417019deb0 con 0x7f41701069f0 2024-03-26T04:12:30.827 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:29.818+0000 7f416d7fa700 1 -- 172.21.15.16:0/629545808 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 848163187 0 0) 0x7f4160004990 con 0x7f41701069f0 2024-03-26T04:12:30.827 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:29.818+0000 7f416d7fa700 1 -- 172.21.15.16:0/629545808 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f4160006110 con 0x7f41701069f0 2024-03-26T04:12:30.827 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:29.818+0000 7f416d7fa700 1 -- 172.21.15.16:0/629545808 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 8) v1 ==== 45893+0+0 (unknown 3089913169 0 0) 0x7f41600115c0 con 0x7f41701069f0 2024-03-26T04:12:30.827 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:29.818+0000 7f416d7fa700 1 -- 172.21.15.16:0/629545808 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (unknown 883473703 0 0) 0x7f41600480e0 con 0x7f41701069f0 2024-03-26T04:12:30.827 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:29.819+0000 7f4176617700 1 -- 172.21.15.16:0/629545808 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4150005120 con 0x7f41701069f0 2024-03-26T04:12:30.827 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:29.824+0000 7f416d7fa700 1 -- 172.21.15.16:0/629545808 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f4160012e20 con 0x7f41701069f0 2024-03-26T04:12:30.827 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:30.053+0000 7f4176617700 1 -- 172.21.15.16:0/629545808 --> v1:172.21.15.16:6800/1828954920 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f4150000b90 con 0x7f415c03a210 2024-03-26T04:12:30.828 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:30.057+0000 7f416d7fa700 1 -- 172.21.15.16:0/629545808 <== mgr.14120 v1:172.21.15.16:6800/1828954920 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (unknown 0 0 3265049985) 0x7f4150000b90 con 0x7f415c03a210 2024-03-26T04:12:30.828 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:30.059+0000 7f4176617700 1 -- 172.21.15.16:0/629545808 >> v1:172.21.15.16:6800/1828954920 conn(0x7f415c03a210 legacy=0x7f415c03c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:30.828 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:30.059+0000 7f4176617700 1 -- 172.21.15.16:0/629545808 >> v1:172.21.15.16:6789/0 conn(0x7f41701069f0 legacy=0x7f4170199810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:30.828 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:30.059+0000 7f4176617700 1 -- 172.21.15.16:0/629545808 shutdown_connections 2024-03-26T04:12:30.828 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:30.059+0000 7f4176617700 1 -- 172.21.15.16:0/629545808 >> 172.21.15.16:0/629545808 conn(0x7f4170100360 msgr2=0x7f4170190650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:30.828 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:30.059+0000 7f4176617700 1 -- 172.21.15.16:0/629545808 shutdown_connections 2024-03-26T04:12:30.828 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:30.059+0000 7f4176617700 1 -- 172.21.15.16:0/629545808 wait complete. 2024-03-26T04:12:30.828 INFO:teuthology.orchestra.run.smithi016.stdout:Deploying unmanaged mgr service... 2024-03-26T04:12:31.068 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:31 smithi016 ceph-mon[29101]: from='client.14142 v1:172.21.15.16:0/629545808' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:12:31.068 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:31 smithi016 ceph-mon[29101]: Saving service mon spec with placement count:5 2024-03-26T04:12:31.068 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:31 smithi016 ceph-mon[29101]: from='mgr.14120 v1:172.21.15.16:0/3870844279' entity='mgr.y' 2024-03-26T04:12:32.041 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-26T04:12:32.041 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:31.248+0000 7f33125ef700 1 Processor -- start 2024-03-26T04:12:32.041 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:31.248+0000 7f33125ef700 1 -- start start 2024-03-26T04:12:32.041 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:31.249+0000 7f33125ef700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f330c10a350 con 0x7f330c105f30 2024-03-26T04:12:32.041 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:31.249+0000 7f330bfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f330c105f30 0x7f330c106350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54308/0 (socket says 172.21.15.16:54308) 2024-03-26T04:12:32.041 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:31.249+0000 7f330bfff700 1 -- 172.21.15.16:0/2585784015 learned_addr learned my addr 172.21.15.16:0/2585784015 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:32.042 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:31.249+0000 7f330affd700 1 -- 172.21.15.16:0/2585784015 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2588812094 0 0) 0x7f330c10a350 con 0x7f330c105f30 2024-03-26T04:12:32.042 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:31.249+0000 7f330affd700 1 -- 172.21.15.16:0/2585784015 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32f80035b0 con 0x7f330c105f30 2024-03-26T04:12:32.042 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:31.249+0000 7f330affd700 1 -- 172.21.15.16:0/2585784015 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 505419005 0 0) 0x7f32f80035b0 con 0x7f330c105f30 2024-03-26T04:12:32.042 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:31.249+0000 7f330affd700 1 -- 172.21.15.16:0/2585784015 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f330c10b4f0 con 0x7f330c105f30 2024-03-26T04:12:32.042 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:31.249+0000 7f330affd700 1 -- 172.21.15.16:0/2585784015 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f32fc001580 con 0x7f330c105f30 2024-03-26T04:12:32.042 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:31.249+0000 7f330affd700 1 -- 172.21.15.16:0/2585784015 <== mon.0 v1:172.21.15.16:6789/0 4 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 848163187 0 0) 0x7f32fc001ce0 con 0x7f330c105f30 2024-03-26T04:12:32.042 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:31.250+0000 7f330affd700 1 -- 172.21.15.16:0/2585784015 <== mon.0 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f32fc005960 con 0x7f330c105f30 2024-03-26T04:12:32.042 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:31.250+0000 7f33125ef700 1 -- 172.21.15.16:0/2585784015 >> v1:172.21.15.16:6789/0 conn(0x7f330c105f30 legacy=0x7f330c106350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:32.042 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:31.250+0000 7f33125ef700 1 -- 172.21.15.16:0/2585784015 shutdown_connections 2024-03-26T04:12:32.042 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:31.250+0000 7f33125ef700 1 -- 172.21.15.16:0/2585784015 >> 172.21.15.16:0/2585784015 conn(0x7f330c1014f0 msgr2=0x7f330c103910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:32.042 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:31.250+0000 7f33125ef700 1 -- 172.21.15.16:0/2585784015 shutdown_connections 2024-03-26T04:12:32.042 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:31.250+0000 7f33125ef700 1 -- 172.21.15.16:0/2585784015 wait complete. 2024-03-26T04:12:32.042 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:31.251+0000 7f33125ef700 1 Processor -- start 2024-03-26T04:12:32.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:31.251+0000 7f33125ef700 1 -- start start 2024-03-26T04:12:32.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:31.251+0000 7f33125ef700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f330c19c0d0 con 0x7f330c105f30 2024-03-26T04:12:32.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:31.251+0000 7f330bfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f330c105f30 0x7f330c19b9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54314/0 (socket says 172.21.15.16:54314) 2024-03-26T04:12:32.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:31.251+0000 7f330bfff700 1 -- 172.21.15.16:0/1103861719 learned_addr learned my addr 172.21.15.16:0/1103861719 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:32.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:31.251+0000 7f33097fa700 1 -- 172.21.15.16:0/1103861719 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3403063609 0 0) 0x7f330c19c0d0 con 0x7f330c105f30 2024-03-26T04:12:32.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:31.251+0000 7f33097fa700 1 -- 172.21.15.16:0/1103861719 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32f80055b0 con 0x7f330c105f30 2024-03-26T04:12:32.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:31.252+0000 7f33097fa700 1 -- 172.21.15.16:0/1103861719 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1900985391 0 0) 0x7f32f80055b0 con 0x7f330c105f30 2024-03-26T04:12:32.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:31.252+0000 7f33097fa700 1 -- 172.21.15.16:0/1103861719 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f330c19c0d0 con 0x7f330c105f30 2024-03-26T04:12:32.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:31.252+0000 7f33097fa700 1 -- 172.21.15.16:0/1103861719 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f32fc004870 con 0x7f330c105f30 2024-03-26T04:12:32.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:31.252+0000 7f33097fa700 1 -- 172.21.15.16:0/1103861719 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4034855931 0 0) 0x7f330c19c0d0 con 0x7f330c105f30 2024-03-26T04:12:32.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:31.252+0000 7f33097fa700 1 -- 172.21.15.16:0/1103861719 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f330c19c260 con 0x7f330c105f30 2024-03-26T04:12:32.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:31.252+0000 7f33125ef700 1 -- 172.21.15.16:0/1103861719 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f330c19c510 con 0x7f330c105f30 2024-03-26T04:12:32.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:31.252+0000 7f33125ef700 1 -- 172.21.15.16:0/1103861719 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f330c1a0060 con 0x7f330c105f30 2024-03-26T04:12:32.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:31.252+0000 7f33097fa700 1 -- 172.21.15.16:0/1103861719 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1037+0+0 (unknown 848163187 0 0) 0x7f32fc004a40 con 0x7f330c105f30 2024-03-26T04:12:32.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:31.252+0000 7f33097fa700 1 -- 172.21.15.16:0/1103861719 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f32fc006070 con 0x7f330c105f30 2024-03-26T04:12:32.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:31.253+0000 7f33097fa700 1 -- 172.21.15.16:0/1103861719 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 8) v1 ==== 45893+0+0 (unknown 3089913169 0 0) 0x7f32fc011520 con 0x7f330c105f30 2024-03-26T04:12:32.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:31.253+0000 7f33097fa700 1 -- 172.21.15.16:0/1103861719 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (unknown 883473703 0 0) 0x7f32fc047130 con 0x7f330c105f30 2024-03-26T04:12:32.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:31.253+0000 7f33125ef700 1 -- 172.21.15.16:0/1103861719 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32ec005150 con 0x7f330c105f30 2024-03-26T04:12:32.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:31.258+0000 7f33097fa700 1 -- 172.21.15.16:0/1103861719 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f32fc012d80 con 0x7f330c105f30 2024-03-26T04:12:32.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:31.495+0000 7f33125ef700 1 -- 172.21.15.16:0/1103861719 --> v1:172.21.15.16:6800/1828954920 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f32ec000bc0 con 0x7f32f803a260 2024-03-26T04:12:32.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:31.499+0000 7f33097fa700 1 -- 172.21.15.16:0/1103861719 <== mgr.14120 v1:172.21.15.16:6800/1828954920 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (unknown 0 0 325935098) 0x7f32ec000bc0 con 0x7f32f803a260 2024-03-26T04:12:32.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:31.501+0000 7f33125ef700 1 -- 172.21.15.16:0/1103861719 >> v1:172.21.15.16:6800/1828954920 conn(0x7f32f803a260 legacy=0x7f32f803c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:32.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:31.501+0000 7f33125ef700 1 -- 172.21.15.16:0/1103861719 >> v1:172.21.15.16:6789/0 conn(0x7f330c105f30 legacy=0x7f330c19b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:32.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:31.501+0000 7f33125ef700 1 -- 172.21.15.16:0/1103861719 shutdown_connections 2024-03-26T04:12:32.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:31.501+0000 7f33125ef700 1 -- 172.21.15.16:0/1103861719 >> 172.21.15.16:0/1103861719 conn(0x7f330c1014f0 msgr2=0x7f330c1018d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:32.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:31.502+0000 7f33125ef700 1 -- 172.21.15.16:0/1103861719 shutdown_connections 2024-03-26T04:12:32.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:31.502+0000 7f33125ef700 1 -- 172.21.15.16:0/1103861719 wait complete. 2024-03-26T04:12:32.891 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:32 smithi016 ceph-mon[29101]: from='client.14144 v1:172.21.15.16:0/1103861719' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:12:32.891 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:32 smithi016 ceph-mon[29101]: Saving service mgr spec with placement count:2 2024-03-26T04:12:32.891 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:32 smithi016 ceph-mon[29101]: from='mgr.14120 v1:172.21.15.16:0/3870844279' entity='mgr.y' 2024-03-26T04:12:34.020 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:33 smithi016 ceph-mon[29101]: from='mgr.14120 v1:172.21.15.16:0/3870844279' entity='mgr.y' 2024-03-26T04:12:34.021 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:33 smithi016 ceph-mon[29101]: from='mgr.14120 v1:172.21.15.16:0/3870844279' entity='mgr.y' 2024-03-26T04:12:34.021 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:33 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2637271223' entity='client.admin' 2024-03-26T04:12:34.039 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:33.215+0000 7f3345964700 1 Processor -- start 2024-03-26T04:12:34.039 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:33.215+0000 7f3345964700 1 -- start start 2024-03-26T04:12:34.040 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:33.216+0000 7f3345964700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3340078ec0 con 0x7f3340106990 2024-03-26T04:12:34.040 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:33.216+0000 7f333effd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3340106990 0x7f3340108d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54328/0 (socket says 172.21.15.16:54328) 2024-03-26T04:12:34.040 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:33.216+0000 7f333effd700 1 -- 172.21.15.16:0/2203467882 learned_addr learned my addr 172.21.15.16:0/2203467882 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:34.040 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:33.216+0000 7f333dffb700 1 -- 172.21.15.16:0/2203467882 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3355530133 0 0) 0x7f3340078ec0 con 0x7f3340106990 2024-03-26T04:12:34.040 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:33.216+0000 7f333dffb700 1 -- 172.21.15.16:0/2203467882 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33280035b0 con 0x7f3340106990 2024-03-26T04:12:34.040 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:33.216+0000 7f333dffb700 1 -- 172.21.15.16:0/2203467882 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1503689153 0 0) 0x7f33280035b0 con 0x7f3340106990 2024-03-26T04:12:34.041 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:33.216+0000 7f333dffb700 1 -- 172.21.15.16:0/2203467882 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f334010b4b0 con 0x7f3340106990 2024-03-26T04:12:34.041 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:33.216+0000 7f333dffb700 1 -- 172.21.15.16:0/2203467882 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f3330001580 con 0x7f3340106990 2024-03-26T04:12:34.041 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:33.217+0000 7f333dffb700 1 -- 172.21.15.16:0/2203467882 <== mon.0 v1:172.21.15.16:6789/0 4 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f3330001d00 con 0x7f3340106990 2024-03-26T04:12:34.041 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:33.217+0000 7f333dffb700 1 -- 172.21.15.16:0/2203467882 <== mon.0 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f3330005960 con 0x7f3340106990 2024-03-26T04:12:34.041 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:33.217+0000 7f3345964700 1 -- 172.21.15.16:0/2203467882 >> v1:172.21.15.16:6789/0 conn(0x7f3340106990 legacy=0x7f3340108d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:34.041 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:33.217+0000 7f3345964700 1 -- 172.21.15.16:0/2203467882 shutdown_connections 2024-03-26T04:12:34.041 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:33.217+0000 7f3345964700 1 -- 172.21.15.16:0/2203467882 >> 172.21.15.16:0/2203467882 conn(0x7f33401003c0 msgr2=0x7f3340102800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:34.042 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:33.217+0000 7f3345964700 1 -- 172.21.15.16:0/2203467882 shutdown_connections 2024-03-26T04:12:34.042 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:33.218+0000 7f3345964700 1 -- 172.21.15.16:0/2203467882 wait complete. 2024-03-26T04:12:34.042 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:33.218+0000 7f3345964700 1 Processor -- start 2024-03-26T04:12:34.042 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:33.218+0000 7f3345964700 1 -- start start 2024-03-26T04:12:34.042 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:33.218+0000 7f3345964700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3340199f70 con 0x7f3340106990 2024-03-26T04:12:34.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:33.218+0000 7f333effd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3340106990 0x7f3340199860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54338/0 (socket says 172.21.15.16:54338) 2024-03-26T04:12:34.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:33.218+0000 7f333effd700 1 -- 172.21.15.16:0/2637271223 learned_addr learned my addr 172.21.15.16:0/2637271223 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:34.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:33.219+0000 7f3344962700 1 -- 172.21.15.16:0/2637271223 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2849621480 0 0) 0x7f3340199f70 con 0x7f3340106990 2024-03-26T04:12:34.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:33.219+0000 7f3344962700 1 -- 172.21.15.16:0/2637271223 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33280055b0 con 0x7f3340106990 2024-03-26T04:12:34.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:33.219+0000 7f3344962700 1 -- 172.21.15.16:0/2637271223 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2300236112 0 0) 0x7f33280055b0 con 0x7f3340106990 2024-03-26T04:12:34.043 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:33.219+0000 7f3344962700 1 -- 172.21.15.16:0/2637271223 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3340199f70 con 0x7f3340106990 2024-03-26T04:12:34.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:33.219+0000 7f3344962700 1 -- 172.21.15.16:0/2637271223 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f3330001080 con 0x7f3340106990 2024-03-26T04:12:34.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:33.219+0000 7f3344962700 1 -- 172.21.15.16:0/2637271223 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2245774805 0 0) 0x7f3340199f70 con 0x7f3340106990 2024-03-26T04:12:34.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:33.219+0000 7f3344962700 1 -- 172.21.15.16:0/2637271223 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f334019a100 con 0x7f3340106990 2024-03-26T04:12:34.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:33.219+0000 7f3345964700 1 -- 172.21.15.16:0/2637271223 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f334019a3b0 con 0x7f3340106990 2024-03-26T04:12:34.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:33.219+0000 7f3345964700 1 -- 172.21.15.16:0/2637271223 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f334019df00 con 0x7f3340106990 2024-03-26T04:12:34.044 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:33.219+0000 7f3344962700 1 -- 172.21.15.16:0/2637271223 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f3330001870 con 0x7f3340106990 2024-03-26T04:12:34.045 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:33.219+0000 7f3344962700 1 -- 172.21.15.16:0/2637271223 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f33300061c0 con 0x7f3340106990 2024-03-26T04:12:34.045 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:33.220+0000 7f3344962700 1 -- 172.21.15.16:0/2637271223 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 8) v1 ==== 45893+0+0 (unknown 3089913169 0 0) 0x7f3330011750 con 0x7f3340106990 2024-03-26T04:12:34.045 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:33.220+0000 7f3344962700 1 -- 172.21.15.16:0/2637271223 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (unknown 883473703 0 0) 0x7f3330048280 con 0x7f3340106990 2024-03-26T04:12:34.045 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:33.220+0000 7f3345964700 1 -- 172.21.15.16:0/2637271223 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f332c005150 con 0x7f3340106990 2024-03-26T04:12:34.045 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:33.225+0000 7f3344962700 1 -- 172.21.15.16:0/2637271223 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f3330012fc0 con 0x7f3340106990 2024-03-26T04:12:34.045 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:33.432+0000 7f3345964700 1 -- 172.21.15.16:0/2637271223 --> v1:172.21.15.16:6789/0 -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f332c005400 con 0x7f3340106990 2024-03-26T04:12:34.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:33.435+0000 7f3344962700 1 -- 172.21.15.16:0/2637271223 <== mon.0 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (unknown 2684808024 0 0) 0x7f33300150a0 con 0x7f3340106990 2024-03-26T04:12:34.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:33.438+0000 7f3345964700 1 -- 172.21.15.16:0/2637271223 >> v1:172.21.15.16:6800/1828954920 conn(0x7f332803a1a0 legacy=0x7f332803c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:34.046 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:33.438+0000 7f3345964700 1 -- 172.21.15.16:0/2637271223 >> v1:172.21.15.16:6789/0 conn(0x7f3340106990 legacy=0x7f3340199860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:34.047 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:33.438+0000 7f3345964700 1 -- 172.21.15.16:0/2637271223 shutdown_connections 2024-03-26T04:12:34.047 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:33.438+0000 7f3345964700 1 -- 172.21.15.16:0/2637271223 >> 172.21.15.16:0/2637271223 conn(0x7f33401003c0 msgr2=0x7f3340190690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:34.047 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:33.438+0000 7f3345964700 1 -- 172.21.15.16:0/2637271223 shutdown_connections 2024-03-26T04:12:34.047 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:33.438+0000 7f3345964700 1 -- 172.21.15.16:0/2637271223 wait complete. 2024-03-26T04:12:35.888 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:35 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/692627902' entity='client.admin' 2024-03-26T04:12:35.889 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:35 smithi016 ceph-mon[29101]: from='mgr.14120 v1:172.21.15.16:0/3870844279' entity='mgr.y' 2024-03-26T04:12:35.899 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:34.450+0000 7f6b7e50c700 1 Processor -- start 2024-03-26T04:12:35.899 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:34.450+0000 7f6b7e50c700 1 -- start start 2024-03-26T04:12:35.900 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:34.450+0000 7f6b7e50c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6b7807a340 con 0x7f6b7807b740 2024-03-26T04:12:35.900 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:34.451+0000 7f6b77fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6b7807b740 0x7f6b78079ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54346/0 (socket says 172.21.15.16:54346) 2024-03-26T04:12:35.900 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:34.451+0000 7f6b77fff700 1 -- 172.21.15.16:0/397861144 learned_addr learned my addr 172.21.15.16:0/397861144 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:35.900 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:34.451+0000 7f6b76ffd700 1 -- 172.21.15.16:0/397861144 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1717357869 0 0) 0x7f6b7807a340 con 0x7f6b7807b740 2024-03-26T04:12:35.900 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:34.451+0000 7f6b76ffd700 1 -- 172.21.15.16:0/397861144 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b600035b0 con 0x7f6b7807b740 2024-03-26T04:12:35.900 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:34.451+0000 7f6b76ffd700 1 -- 172.21.15.16:0/397861144 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3262843576 0 0) 0x7f6b600035b0 con 0x7f6b7807b740 2024-03-26T04:12:35.901 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:34.451+0000 7f6b76ffd700 1 -- 172.21.15.16:0/397861144 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b7807a4d0 con 0x7f6b7807b740 2024-03-26T04:12:35.901 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:34.451+0000 7f6b76ffd700 1 -- 172.21.15.16:0/397861144 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f6b68001580 con 0x7f6b7807b740 2024-03-26T04:12:35.901 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:34.451+0000 7f6b76ffd700 1 -- 172.21.15.16:0/397861144 <== mon.0 v1:172.21.15.16:6789/0 4 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f6b68001d00 con 0x7f6b7807b740 2024-03-26T04:12:35.901 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:34.451+0000 7f6b76ffd700 1 -- 172.21.15.16:0/397861144 <== mon.0 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f6b68005960 con 0x7f6b7807b740 2024-03-26T04:12:35.902 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:34.452+0000 7f6b7e50c700 1 -- 172.21.15.16:0/397861144 >> v1:172.21.15.16:6789/0 conn(0x7f6b7807b740 legacy=0x7f6b78079ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:35.902 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:34.452+0000 7f6b7e50c700 1 -- 172.21.15.16:0/397861144 shutdown_connections 2024-03-26T04:12:35.902 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:34.452+0000 7f6b7e50c700 1 -- 172.21.15.16:0/397861144 >> 172.21.15.16:0/397861144 conn(0x7f6b78101460 msgr2=0x7f6b781038e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:35.902 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:34.452+0000 7f6b7e50c700 1 -- 172.21.15.16:0/397861144 shutdown_connections 2024-03-26T04:12:35.902 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:34.452+0000 7f6b7e50c700 1 -- 172.21.15.16:0/397861144 wait complete. 2024-03-26T04:12:35.902 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:34.452+0000 7f6b7e50c700 1 Processor -- start 2024-03-26T04:12:35.903 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:34.452+0000 7f6b7e50c700 1 -- start start 2024-03-26T04:12:35.903 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:34.453+0000 7f6b7e50c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6b7819c100 con 0x7f6b7807b740 2024-03-26T04:12:35.903 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:34.453+0000 7f6b77fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6b7807b740 0x7f6b7819b9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54354/0 (socket says 172.21.15.16:54354) 2024-03-26T04:12:35.903 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:34.453+0000 7f6b77fff700 1 -- 172.21.15.16:0/692627902 learned_addr learned my addr 172.21.15.16:0/692627902 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:35.903 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:34.453+0000 7f6b757fa700 1 -- 172.21.15.16:0/692627902 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2714378083 0 0) 0x7f6b7819c100 con 0x7f6b7807b740 2024-03-26T04:12:35.903 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:34.453+0000 7f6b757fa700 1 -- 172.21.15.16:0/692627902 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b600055b0 con 0x7f6b7807b740 2024-03-26T04:12:35.904 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:34.453+0000 7f6b757fa700 1 -- 172.21.15.16:0/692627902 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1571475663 0 0) 0x7f6b600055b0 con 0x7f6b7807b740 2024-03-26T04:12:35.904 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:34.453+0000 7f6b757fa700 1 -- 172.21.15.16:0/692627902 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b7819c100 con 0x7f6b7807b740 2024-03-26T04:12:35.904 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:34.453+0000 7f6b757fa700 1 -- 172.21.15.16:0/692627902 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f6b68001080 con 0x7f6b7807b740 2024-03-26T04:12:35.904 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:34.453+0000 7f6b757fa700 1 -- 172.21.15.16:0/692627902 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2138150558 0 0) 0x7f6b7819c100 con 0x7f6b7807b740 2024-03-26T04:12:35.904 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:34.454+0000 7f6b757fa700 1 -- 172.21.15.16:0/692627902 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b7819c290 con 0x7f6b7807b740 2024-03-26T04:12:35.904 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:34.454+0000 7f6b7e50c700 1 -- 172.21.15.16:0/692627902 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b7819c540 con 0x7f6b7807b740 2024-03-26T04:12:35.904 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:34.454+0000 7f6b7e50c700 1 -- 172.21.15.16:0/692627902 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b781a0090 con 0x7f6b7807b740 2024-03-26T04:12:35.905 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:34.454+0000 7f6b757fa700 1 -- 172.21.15.16:0/692627902 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f6b68004a60 con 0x7f6b7807b740 2024-03-26T04:12:35.905 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:34.454+0000 7f6b757fa700 1 -- 172.21.15.16:0/692627902 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f6b68006280 con 0x7f6b7807b740 2024-03-26T04:12:35.905 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:34.454+0000 7f6b757fa700 1 -- 172.21.15.16:0/692627902 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 8) v1 ==== 45893+0+0 (unknown 3089913169 0 0) 0x7f6b68011730 con 0x7f6b7807b740 2024-03-26T04:12:35.905 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:34.454+0000 7f6b7e50c700 1 -- 172.21.15.16:0/692627902 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b64005150 con 0x7f6b7807b740 2024-03-26T04:12:35.905 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:34.455+0000 7f6b757fa700 1 -- 172.21.15.16:0/692627902 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (unknown 883473703 0 0) 0x7f6b68047200 con 0x7f6b7807b740 2024-03-26T04:12:35.905 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:34.459+0000 7f6b757fa700 1 -- 172.21.15.16:0/692627902 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f6b68012e60 con 0x7f6b7807b740 2024-03-26T04:12:35.906 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:34.675+0000 7f6b7e50c700 1 -- 172.21.15.16:0/692627902 --> v1:172.21.15.16:6789/0 -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f6b64005980 con 0x7f6b7807b740 2024-03-26T04:12:35.906 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:34.677+0000 7f6b757fa700 1 -- 172.21.15.16:0/692627902 <== mon.0 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (unknown 1496729260 0 0) 0x7f6b68014f40 con 0x7f6b7807b740 2024-03-26T04:12:35.906 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:34.680+0000 7f6b7e50c700 1 -- 172.21.15.16:0/692627902 >> v1:172.21.15.16:6800/1828954920 conn(0x7f6b6003a230 legacy=0x7f6b6003c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:35.906 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:34.680+0000 7f6b7e50c700 1 -- 172.21.15.16:0/692627902 >> v1:172.21.15.16:6789/0 conn(0x7f6b7807b740 legacy=0x7f6b7819b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:35.906 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:34.680+0000 7f6b7e50c700 1 -- 172.21.15.16:0/692627902 shutdown_connections 2024-03-26T04:12:35.906 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:34.680+0000 7f6b7e50c700 1 -- 172.21.15.16:0/692627902 >> 172.21.15.16:0/692627902 conn(0x7f6b78101460 msgr2=0x7f6b7810c5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:35.907 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:34.680+0000 7f6b7e50c700 1 -- 172.21.15.16:0/692627902 shutdown_connections 2024-03-26T04:12:35.907 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:34.680+0000 7f6b7e50c700 1 -- 172.21.15.16:0/692627902 wait complete. 2024-03-26T04:12:35.907 INFO:teuthology.orchestra.run.smithi016.stdout:Enabling the dashboard module... 2024-03-26T04:12:36.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:36 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/931749062' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-26T04:12:36.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:36 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ignoring --setuser ceph since I am not root 2024-03-26T04:12:36.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:36 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ignoring --setgroup ceph since I am not root 2024-03-26T04:12:37.286 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.358+0000 7fcc4824a700 1 Processor -- start 2024-03-26T04:12:37.286 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.358+0000 7fcc4824a700 1 -- start start 2024-03-26T04:12:37.286 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.358+0000 7fcc4824a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcc40079750 con 0x7fcc401069d0 2024-03-26T04:12:37.286 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.358+0000 7fcc45fe6700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fcc401069d0 0x7fcc40108dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54758/0 (socket says 172.21.15.16:54758) 2024-03-26T04:12:37.287 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.358+0000 7fcc45fe6700 1 -- 172.21.15.16:0/3187020962 learned_addr learned my addr 172.21.15.16:0/3187020962 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:37.287 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.359+0000 7fcc44fe4700 1 -- 172.21.15.16:0/3187020962 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2363729152 0 0) 0x7fcc40079750 con 0x7fcc401069d0 2024-03-26T04:12:37.287 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.359+0000 7fcc44fe4700 1 -- 172.21.15.16:0/3187020962 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc2c0035b0 con 0x7fcc401069d0 2024-03-26T04:12:37.287 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.359+0000 7fcc44fe4700 1 -- 172.21.15.16:0/3187020962 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3874958914 0 0) 0x7fcc2c0035b0 con 0x7fcc401069d0 2024-03-26T04:12:37.287 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.359+0000 7fcc44fe4700 1 -- 172.21.15.16:0/3187020962 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc4010b4f0 con 0x7fcc401069d0 2024-03-26T04:12:37.287 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.359+0000 7fcc44fe4700 1 -- 172.21.15.16:0/3187020962 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7fcc3c001580 con 0x7fcc401069d0 2024-03-26T04:12:37.287 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.359+0000 7fcc44fe4700 1 -- 172.21.15.16:0/3187020962 <== mon.0 v1:172.21.15.16:6789/0 4 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7fcc3c001c30 con 0x7fcc401069d0 2024-03-26T04:12:37.287 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.359+0000 7fcc44fe4700 1 -- 172.21.15.16:0/3187020962 <== mon.0 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7fcc3c0057e0 con 0x7fcc401069d0 2024-03-26T04:12:37.288 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.360+0000 7fcc4824a700 1 -- 172.21.15.16:0/3187020962 >> v1:172.21.15.16:6789/0 conn(0x7fcc401069d0 legacy=0x7fcc40108dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:37.288 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.360+0000 7fcc4824a700 1 -- 172.21.15.16:0/3187020962 shutdown_connections 2024-03-26T04:12:37.288 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.360+0000 7fcc4824a700 1 -- 172.21.15.16:0/3187020962 >> 172.21.15.16:0/3187020962 conn(0x7fcc40100420 msgr2=0x7fcc40102840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:37.288 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.360+0000 7fcc4824a700 1 -- 172.21.15.16:0/3187020962 shutdown_connections 2024-03-26T04:12:37.288 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.360+0000 7fcc4824a700 1 -- 172.21.15.16:0/3187020962 wait complete. 2024-03-26T04:12:37.288 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.361+0000 7fcc4824a700 1 Processor -- start 2024-03-26T04:12:37.288 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.361+0000 7fcc4824a700 1 -- start start 2024-03-26T04:12:37.288 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.361+0000 7fcc4824a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fcc4019f8d0 con 0x7fcc401069d0 2024-03-26T04:12:37.288 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.361+0000 7fcc45fe6700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fcc401069d0 0x7fcc40106250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54770/0 (socket says 172.21.15.16:54770) 2024-03-26T04:12:37.288 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.361+0000 7fcc45fe6700 1 -- 172.21.15.16:0/931749062 learned_addr learned my addr 172.21.15.16:0/931749062 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:37.289 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.361+0000 7fcc36ffd700 1 -- 172.21.15.16:0/931749062 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4243579368 0 0) 0x7fcc4019f8d0 con 0x7fcc401069d0 2024-03-26T04:12:37.289 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.361+0000 7fcc36ffd700 1 -- 172.21.15.16:0/931749062 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc2c005600 con 0x7fcc401069d0 2024-03-26T04:12:37.289 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.361+0000 7fcc36ffd700 1 -- 172.21.15.16:0/931749062 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2598287411 0 0) 0x7fcc2c005600 con 0x7fcc401069d0 2024-03-26T04:12:37.289 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.361+0000 7fcc36ffd700 1 -- 172.21.15.16:0/931749062 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc4019f8d0 con 0x7fcc401069d0 2024-03-26T04:12:37.289 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.361+0000 7fcc36ffd700 1 -- 172.21.15.16:0/931749062 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7fcc3c000ed0 con 0x7fcc401069d0 2024-03-26T04:12:37.289 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.362+0000 7fcc36ffd700 1 -- 172.21.15.16:0/931749062 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4080408187 0 0) 0x7fcc4019f8d0 con 0x7fcc401069d0 2024-03-26T04:12:37.289 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.362+0000 7fcc36ffd700 1 -- 172.21.15.16:0/931749062 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc401a0a70 con 0x7fcc401069d0 2024-03-26T04:12:37.289 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.362+0000 7fcc4824a700 1 -- 172.21.15.16:0/931749062 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc4019fac0 con 0x7fcc401069d0 2024-03-26T04:12:37.289 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.362+0000 7fcc4824a700 1 -- 172.21.15.16:0/931749062 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc4019ffc0 con 0x7fcc401069d0 2024-03-26T04:12:37.290 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.362+0000 7fcc36ffd700 1 -- 172.21.15.16:0/931749062 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7fcc3c0049d0 con 0x7fcc401069d0 2024-03-26T04:12:37.290 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.362+0000 7fcc36ffd700 1 -- 172.21.15.16:0/931749062 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7fcc3c0060c0 con 0x7fcc401069d0 2024-03-26T04:12:37.290 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.363+0000 7fcc36ffd700 1 -- 172.21.15.16:0/931749062 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 8) v1 ==== 45893+0+0 (unknown 3089913169 0 0) 0x7fcc3c011590 con 0x7fcc401069d0 2024-03-26T04:12:37.290 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.363+0000 7fcc4824a700 1 -- 172.21.15.16:0/931749062 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc24005150 con 0x7fcc401069d0 2024-03-26T04:12:37.290 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.363+0000 7fcc36ffd700 1 -- 172.21.15.16:0/931749062 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (unknown 883473703 0 0) 0x7fcc3c047d70 con 0x7fcc401069d0 2024-03-26T04:12:37.290 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.368+0000 7fcc36ffd700 1 -- 172.21.15.16:0/931749062 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fcc3c013010 con 0x7fcc401069d0 2024-03-26T04:12:37.290 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.630+0000 7fcc4824a700 1 -- 172.21.15.16:0/931749062 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fcc24005400 con 0x7fcc401069d0 2024-03-26T04:12:37.290 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.679+0000 7fcc36ffd700 1 -- 172.21.15.16:0/931749062 <== mon.0 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (unknown 1498667528 0 0) 0x7fcc3c014f40 con 0x7fcc401069d0 2024-03-26T04:12:37.290 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.681+0000 7fcc36ffd700 1 -- 172.21.15.16:0/931749062 <== mon.0 v1:172.21.15.16:6789/0 11 ==== mgrmap(e 9) v1 ==== 45906+0+0 (unknown 224057423 0 0) 0x7fcc3c046ac0 con 0x7fcc401069d0 2024-03-26T04:12:37.290 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.683+0000 7fcc4824a700 1 -- 172.21.15.16:0/931749062 >> v1:172.21.15.16:6800/1828954920 conn(0x7fcc2c03a220 legacy=0x7fcc2c03c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:37.290 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.683+0000 7fcc4824a700 1 -- 172.21.15.16:0/931749062 >> v1:172.21.15.16:6789/0 conn(0x7fcc401069d0 legacy=0x7fcc40106250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:37.290 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.683+0000 7fcc4824a700 1 -- 172.21.15.16:0/931749062 shutdown_connections 2024-03-26T04:12:37.291 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.683+0000 7fcc4824a700 1 -- 172.21.15.16:0/931749062 >> 172.21.15.16:0/931749062 conn(0x7fcc40100420 msgr2=0x7fcc401906d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:37.291 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.683+0000 7fcc4824a700 1 -- 172.21.15.16:0/931749062 shutdown_connections 2024-03-26T04:12:37.291 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:36.683+0000 7fcc4824a700 1 -- 172.21.15.16:0/931749062 wait complete. 2024-03-26T04:12:37.520 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:37 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:37.268+0000 7fdf9707a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T04:12:37.520 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:37 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:37.372+0000 7fdf9707a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T04:12:37.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:37 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/931749062' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-26T04:12:37.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:37 smithi016 ceph-mon[29101]: mgrmap e9: y(active, since 19s) 2024-03-26T04:12:38.285 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:38 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:38.045+0000 7fdf9707a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T04:12:38.286 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:38 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:38.140+0000 7fdf9707a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T04:12:38.286 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:38 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:38.285+0000 7fdf9707a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T04:12:38.593 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:38 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:38.384+0000 7fdf9707a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T04:12:38.593 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:38 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:38.470+0000 7fdf9707a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T04:12:38.596 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-03-26T04:12:38.601 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-26T04:12:38.601 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-26T04:12:38.601 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-26T04:12:38.601 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-26T04:12:38.602 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-03-26T04:12:38.602 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.715+0000 7fc3c9377700 1 Processor -- start 2024-03-26T04:12:38.602 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.715+0000 7fc3c9377700 1 -- start start 2024-03-26T04:12:38.602 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.715+0000 7fc3c9377700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc3c407a360 con 0x7fc3c407b760 2024-03-26T04:12:38.602 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.716+0000 7fc3c2ffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc3c407b760 0x7fc3c4079bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54806/0 (socket says 172.21.15.16:54806) 2024-03-26T04:12:38.602 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.716+0000 7fc3c2ffd700 1 -- 172.21.15.16:0/926284644 learned_addr learned my addr 172.21.15.16:0/926284644 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:38.602 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.716+0000 7fc3c1ffb700 1 -- 172.21.15.16:0/926284644 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 798574287 0 0) 0x7fc3c407a360 con 0x7fc3c407b760 2024-03-26T04:12:38.602 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.716+0000 7fc3c1ffb700 1 -- 172.21.15.16:0/926284644 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3b00035b0 con 0x7fc3c407b760 2024-03-26T04:12:38.602 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.716+0000 7fc3c1ffb700 1 -- 172.21.15.16:0/926284644 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2813891513 0 0) 0x7fc3b00035b0 con 0x7fc3c407b760 2024-03-26T04:12:38.602 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.716+0000 7fc3c1ffb700 1 -- 172.21.15.16:0/926284644 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c407a4f0 con 0x7fc3c407b760 2024-03-26T04:12:38.602 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.716+0000 7fc3c1ffb700 1 -- 172.21.15.16:0/926284644 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7fc3b4001580 con 0x7fc3c407b760 2024-03-26T04:12:38.602 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.716+0000 7fc3c1ffb700 1 -- 172.21.15.16:0/926284644 <== mon.0 v1:172.21.15.16:6789/0 4 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7fc3b4001d00 con 0x7fc3c407b760 2024-03-26T04:12:38.602 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.716+0000 7fc3c1ffb700 1 -- 172.21.15.16:0/926284644 <== mon.0 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7fc3b4005960 con 0x7fc3c407b760 2024-03-26T04:12:38.603 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.717+0000 7fc3c9377700 1 -- 172.21.15.16:0/926284644 >> v1:172.21.15.16:6789/0 conn(0x7fc3c407b760 legacy=0x7fc3c4079bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:38.603 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.717+0000 7fc3c9377700 1 -- 172.21.15.16:0/926284644 shutdown_connections 2024-03-26T04:12:38.603 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.717+0000 7fc3c9377700 1 -- 172.21.15.16:0/926284644 >> 172.21.15.16:0/926284644 conn(0x7fc3c4101510 msgr2=0x7fc3c4103990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:38.603 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.717+0000 7fc3c9377700 1 -- 172.21.15.16:0/926284644 shutdown_connections 2024-03-26T04:12:38.603 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.717+0000 7fc3c9377700 1 -- 172.21.15.16:0/926284644 wait complete. 2024-03-26T04:12:38.603 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.718+0000 7fc3c9377700 1 Processor -- start 2024-03-26T04:12:38.603 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.718+0000 7fc3c9377700 1 -- start start 2024-03-26T04:12:38.603 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.718+0000 7fc3c9377700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc3c419c1e0 con 0x7fc3c407b760 2024-03-26T04:12:38.603 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.718+0000 7fc3c2ffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc3c407b760 0x7fc3c419bad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54812/0 (socket says 172.21.15.16:54812) 2024-03-26T04:12:38.603 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.718+0000 7fc3c2ffd700 1 -- 172.21.15.16:0/2878271738 learned_addr learned my addr 172.21.15.16:0/2878271738 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:38.603 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.718+0000 7fc3bbfff700 1 -- 172.21.15.16:0/2878271738 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2873351527 0 0) 0x7fc3c419c1e0 con 0x7fc3c407b760 2024-03-26T04:12:38.603 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.718+0000 7fc3bbfff700 1 -- 172.21.15.16:0/2878271738 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3b00055b0 con 0x7fc3c407b760 2024-03-26T04:12:38.603 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.718+0000 7fc3bbfff700 1 -- 172.21.15.16:0/2878271738 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2625378619 0 0) 0x7fc3b00055b0 con 0x7fc3c407b760 2024-03-26T04:12:38.604 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.718+0000 7fc3bbfff700 1 -- 172.21.15.16:0/2878271738 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc3c419c1e0 con 0x7fc3c407b760 2024-03-26T04:12:38.604 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.719+0000 7fc3bbfff700 1 -- 172.21.15.16:0/2878271738 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7fc3b4001080 con 0x7fc3c407b760 2024-03-26T04:12:38.604 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.719+0000 7fc3bbfff700 1 -- 172.21.15.16:0/2878271738 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1491543662 0 0) 0x7fc3c419c1e0 con 0x7fc3c407b760 2024-03-26T04:12:38.604 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.719+0000 7fc3bbfff700 1 -- 172.21.15.16:0/2878271738 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c419c370 con 0x7fc3c407b760 2024-03-26T04:12:38.604 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.719+0000 7fc3c9377700 1 -- 172.21.15.16:0/2878271738 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3c419c620 con 0x7fc3c407b760 2024-03-26T04:12:38.604 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.719+0000 7fc3c9377700 1 -- 172.21.15.16:0/2878271738 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3c41a0170 con 0x7fc3c407b760 2024-03-26T04:12:38.604 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.719+0000 7fc3bbfff700 1 -- 172.21.15.16:0/2878271738 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7fc3b4004a60 con 0x7fc3c407b760 2024-03-26T04:12:38.604 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.719+0000 7fc3bbfff700 1 -- 172.21.15.16:0/2878271738 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7fc3b4006120 con 0x7fc3c407b760 2024-03-26T04:12:38.604 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.720+0000 7fc3bbfff700 1 -- 172.21.15.16:0/2878271738 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 9) v1 ==== 45906+0+0 (unknown 224057423 0 0) 0x7fc3b4011600 con 0x7fc3c407b760 2024-03-26T04:12:38.604 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.720+0000 7fc3bbfff700 1 -- 172.21.15.16:0/2878271738 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (unknown 883473703 0 0) 0x7fc3b40481d0 con 0x7fc3c407b760 2024-03-26T04:12:38.604 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.720+0000 7fc3c27fc700 1 -- 172.21.15.16:0/2878271738 >> v1:172.21.15.16:6800/1828954920 conn(0x7fc3b003a2b0 legacy=0x7fc3b003c770 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.16:6800/1828954920 2024-03-26T04:12:38.604 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.720+0000 7fc3c9377700 1 -- 172.21.15.16:0/2878271738 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3a4005150 con 0x7fc3c407b760 2024-03-26T04:12:38.604 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.725+0000 7fc3bbfff700 1 -- 172.21.15.16:0/2878271738 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fc3b4012eb0 con 0x7fc3c407b760 2024-03-26T04:12:38.604 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.921+0000 7fc3c27fc700 1 -- 172.21.15.16:0/2878271738 >> v1:172.21.15.16:6800/1828954920 conn(0x7fc3b003a2b0 legacy=0x7fc3b003c770 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.16:6800/1828954920 2024-03-26T04:12:38.605 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.988+0000 7fc3c9377700 1 -- 172.21.15.16:0/2878271738 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fc3a4005c10 con 0x7fc3c407b760 2024-03-26T04:12:38.605 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.988+0000 7fc3bbfff700 1 -- 172.21.15.16:0/2878271738 <== mon.0 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (unknown 2005831594 0 1748205903) 0x7fc3b4014f90 con 0x7fc3c407b760 2024-03-26T04:12:38.605 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.990+0000 7fc3c9377700 1 -- 172.21.15.16:0/2878271738 >> v1:172.21.15.16:6800/1828954920 conn(0x7fc3b003a2b0 legacy=0x7fc3b003c770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T04:12:38.605 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.990+0000 7fc3c9377700 1 -- 172.21.15.16:0/2878271738 >> v1:172.21.15.16:6789/0 conn(0x7fc3c407b760 legacy=0x7fc3c419bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:38.605 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.990+0000 7fc3c9377700 1 -- 172.21.15.16:0/2878271738 shutdown_connections 2024-03-26T04:12:38.605 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.990+0000 7fc3c9377700 1 -- 172.21.15.16:0/2878271738 >> 172.21.15.16:0/2878271738 conn(0x7fc3c4101510 msgr2=0x7fc3c410a3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:38.605 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.990+0000 7fc3c9377700 1 -- 172.21.15.16:0/2878271738 shutdown_connections 2024-03-26T04:12:38.605 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:37.990+0000 7fc3c9377700 1 -- 172.21.15.16:0/2878271738 wait complete. 2024-03-26T04:12:38.605 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for the mgr to restart... 2024-03-26T04:12:38.605 INFO:teuthology.orchestra.run.smithi016.stdout:Waiting for mgr epoch 9... 2024-03-26T04:12:38.861 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:38 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2878271738' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-26T04:12:38.861 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:38 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:38.746+0000 7fdf9707a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T04:12:38.862 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:38 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:38.860+0000 7fdf9707a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T04:12:39.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:38 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:38.953+0000 7fdf9707a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T04:12:39.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:39 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:39.053+0000 7fdf9707a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T04:12:39.662 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:39 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:39.234+0000 7fdf9707a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T04:12:40.487 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:40 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:40.225+0000 7fdf9707a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T04:12:40.487 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:40 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:40.361+0000 7fdf9707a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T04:12:40.793 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:40 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:40.700+0000 7fdf9707a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T04:12:41.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:40 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:40.798+0000 7fdf9707a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T04:12:41.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:41 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:41.002+0000 7fdf9707a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:12:41.496 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:41 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:41.194+0000 7fdf9707a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T04:12:41.758 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:41 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:41.496+0000 7fdf9707a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:12:41.758 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:41 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:41.757+0000 7fdf9707a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T04:12:42.089 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:42 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:42.003+0000 7fdf9707a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T04:12:43.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:42 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:42.839+0000 7fdf9707a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T04:12:43.747 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:43 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:43.421+0000 7fdf9707a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T04:12:43.747 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:43 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:43.523+0000 7fdf9707a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T04:12:43.747 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:12:43 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:12:43.746+0000 7fdf9707a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T04:12:44.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:43 smithi016 ceph-mon[29101]: Active manager daemon y restarted 2024-03-26T04:12:44.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:43 smithi016 ceph-mon[29101]: Activating manager daemon y 2024-03-26T04:12:44.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:43 smithi016 ceph-mon[29101]: osdmap e3: 0 total, 0 up, 0 in 2024-03-26T04:12:44.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:43 smithi016 ceph-mon[29101]: mgrmap e10: y(active, starting, since 0.00322629s) 2024-03-26T04:12:44.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:43 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:12:44.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:43 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T04:12:44.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:43 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T04:12:44.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:43 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T04:12:44.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:43 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T04:12:44.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:43 smithi016 ceph-mon[29101]: Manager daemon y is now available 2024-03-26T04:12:44.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:43 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:12:44.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:43 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:12:45.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:44 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:12:45.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:44 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:12:45.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:44 smithi016 ceph-mon[29101]: [26/Mar/2024:04:12:44] ENGINE Bus STARTING 2024-03-26T04:12:45.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:44 smithi016 ceph-mon[29101]: [26/Mar/2024:04:12:44] ENGINE Serving on https://172.21.15.16:7150 2024-03-26T04:12:45.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:44 smithi016 ceph-mon[29101]: [26/Mar/2024:04:12:44] ENGINE Serving on http://172.21.15.16:8765 2024-03-26T04:12:45.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:44 smithi016 ceph-mon[29101]: [26/Mar/2024:04:12:44] ENGINE Bus STARTED 2024-03-26T04:12:45.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:44 smithi016 ceph-mon[29101]: mgrmap e11: y(active, since 1.00568s) 2024-03-26T04:12:45.618 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout { 2024-03-26T04:12:45.619 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-26T04:12:45.619 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-26T04:12:45.619 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout } 2024-03-26T04:12:45.619 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:39.079+0000 7f5b1b5fb700 1 Processor -- start 2024-03-26T04:12:45.619 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:39.079+0000 7f5b1b5fb700 1 -- start start 2024-03-26T04:12:45.619 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:39.080+0000 7f5b1b5fb700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5b1410a3d0 con 0x7f5b14104210 2024-03-26T04:12:45.619 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:39.080+0000 7f5b19397700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5b14104210 0x7f5b14104630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54822/0 (socket says 172.21.15.16:54822) 2024-03-26T04:12:45.619 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:39.080+0000 7f5b19397700 1 -- 172.21.15.16:0/651716290 learned_addr learned my addr 172.21.15.16:0/651716290 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:45.619 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:39.080+0000 7f5b07fff700 1 -- 172.21.15.16:0/651716290 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2522395338 0 0) 0x7f5b1410a3d0 con 0x7f5b14104210 2024-03-26T04:12:45.619 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:39.080+0000 7f5b07fff700 1 -- 172.21.15.16:0/651716290 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b000035b0 con 0x7f5b14104210 2024-03-26T04:12:45.619 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:39.080+0000 7f5b07fff700 1 -- 172.21.15.16:0/651716290 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1778191628 0 0) 0x7f5b000035b0 con 0x7f5b14104210 2024-03-26T04:12:45.619 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:39.080+0000 7f5b07fff700 1 -- 172.21.15.16:0/651716290 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b1410b570 con 0x7f5b14104210 2024-03-26T04:12:45.619 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:39.080+0000 7f5b07fff700 1 -- 172.21.15.16:0/651716290 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f5b10001580 con 0x7f5b14104210 2024-03-26T04:12:45.619 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:39.081+0000 7f5b07fff700 1 -- 172.21.15.16:0/651716290 <== mon.0 v1:172.21.15.16:6789/0 4 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f5b10001c30 con 0x7f5b14104210 2024-03-26T04:12:45.619 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:39.081+0000 7f5b07fff700 1 -- 172.21.15.16:0/651716290 <== mon.0 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f5b100057e0 con 0x7f5b14104210 2024-03-26T04:12:45.620 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:39.081+0000 7f5b1b5fb700 1 -- 172.21.15.16:0/651716290 >> v1:172.21.15.16:6789/0 conn(0x7f5b14104210 legacy=0x7f5b14104630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:45.620 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:39.081+0000 7f5b1b5fb700 1 -- 172.21.15.16:0/651716290 shutdown_connections 2024-03-26T04:12:45.620 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:39.081+0000 7f5b1b5fb700 1 -- 172.21.15.16:0/651716290 >> 172.21.15.16:0/651716290 conn(0x7f5b140ffb00 msgr2=0x7f5b14101f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:45.620 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:39.081+0000 7f5b1b5fb700 1 -- 172.21.15.16:0/651716290 shutdown_connections 2024-03-26T04:12:45.620 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:39.081+0000 7f5b1b5fb700 1 -- 172.21.15.16:0/651716290 wait complete. 2024-03-26T04:12:45.620 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:39.082+0000 7f5b1b5fb700 1 Processor -- start 2024-03-26T04:12:45.620 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:39.082+0000 7f5b1b5fb700 1 -- start start 2024-03-26T04:12:45.620 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:39.082+0000 7f5b1b5fb700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5b1419f8b0 con 0x7f5b14104210 2024-03-26T04:12:45.620 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:39.082+0000 7f5b19397700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5b14104210 0x7f5b14106a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54836/0 (socket says 172.21.15.16:54836) 2024-03-26T04:12:45.620 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:39.082+0000 7f5b19397700 1 -- 172.21.15.16:0/1565834463 learned_addr learned my addr 172.21.15.16:0/1565834463 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:45.620 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:39.082+0000 7f5b067fc700 1 -- 172.21.15.16:0/1565834463 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2617677265 0 0) 0x7f5b1419f8b0 con 0x7f5b14104210 2024-03-26T04:12:45.621 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:39.082+0000 7f5b067fc700 1 -- 172.21.15.16:0/1565834463 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b00005600 con 0x7f5b14104210 2024-03-26T04:12:45.621 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:39.083+0000 7f5b067fc700 1 -- 172.21.15.16:0/1565834463 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3674392250 0 0) 0x7f5b00005600 con 0x7f5b14104210 2024-03-26T04:12:45.621 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:39.083+0000 7f5b067fc700 1 -- 172.21.15.16:0/1565834463 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b1419f8b0 con 0x7f5b14104210 2024-03-26T04:12:45.621 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:39.083+0000 7f5b067fc700 1 -- 172.21.15.16:0/1565834463 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f5b10000ed0 con 0x7f5b14104210 2024-03-26T04:12:45.621 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:39.083+0000 7f5b067fc700 1 -- 172.21.15.16:0/1565834463 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 935427051 0 0) 0x7f5b1419f8b0 con 0x7f5b14104210 2024-03-26T04:12:45.621 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:39.083+0000 7f5b067fc700 1 -- 172.21.15.16:0/1565834463 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b141a0a50 con 0x7f5b14104210 2024-03-26T04:12:45.621 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:39.083+0000 7f5b1b5fb700 1 -- 172.21.15.16:0/1565834463 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b1419faa0 con 0x7f5b14104210 2024-03-26T04:12:45.621 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:39.083+0000 7f5b1b5fb700 1 -- 172.21.15.16:0/1565834463 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b1419ffa0 con 0x7f5b14104210 2024-03-26T04:12:45.621 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:39.083+0000 7f5b067fc700 1 -- 172.21.15.16:0/1565834463 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f5b100049d0 con 0x7f5b14104210 2024-03-26T04:12:45.621 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:39.083+0000 7f5b067fc700 1 -- 172.21.15.16:0/1565834463 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f5b10006170 con 0x7f5b14104210 2024-03-26T04:12:45.621 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:39.084+0000 7f5b067fc700 1 -- 172.21.15.16:0/1565834463 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 9) v1 ==== 45906+0+0 (unknown 224057423 0 0) 0x7f5b10011630 con 0x7f5b14104210 2024-03-26T04:12:45.621 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:39.084+0000 7f5b067fc700 1 -- 172.21.15.16:0/1565834463 --> v1:172.21.15.16:6800/1828954920 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5b0003df30 con 0x7f5b0003a260 2024-03-26T04:12:45.621 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:39.084+0000 7f5b067fc700 1 -- 172.21.15.16:0/1565834463 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 897+0+0 (unknown 883473703 0 0) 0x7f5b10047680 con 0x7f5b14104210 2024-03-26T04:12:45.621 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:39.084+0000 7f5b18b96700 1 -- 172.21.15.16:0/1565834463 >> v1:172.21.15.16:6800/1828954920 conn(0x7f5b0003a260 legacy=0x7f5b0003c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.16:6800/1828954920 2024-03-26T04:12:45.621 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:39.285+0000 7f5b18b96700 1 -- 172.21.15.16:0/1565834463 >> v1:172.21.15.16:6800/1828954920 conn(0x7f5b0003a260 legacy=0x7f5b0003c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.16:6800/1828954920 2024-03-26T04:12:45.622 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:39.686+0000 7f5b18b96700 1 -- 172.21.15.16:0/1565834463 >> v1:172.21.15.16:6800/1828954920 conn(0x7f5b0003a260 legacy=0x7f5b0003c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.16:6800/1828954920 2024-03-26T04:12:45.622 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:40.487+0000 7f5b18b96700 1 -- 172.21.15.16:0/1565834463 >> v1:172.21.15.16:6800/1828954920 conn(0x7f5b0003a260 legacy=0x7f5b0003c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.16:6800/1828954920 2024-03-26T04:12:45.622 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:42.088+0000 7f5b18b96700 1 -- 172.21.15.16:0/1565834463 >> v1:172.21.15.16:6800/1828954920 conn(0x7f5b0003a260 legacy=0x7f5b0003c720 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.16:6800/1828954920 2024-03-26T04:12:45.622 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:43.761+0000 7f5b067fc700 1 -- 172.21.15.16:0/1565834463 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mgrmap(e 10) v1 ==== 45708+0+0 (unknown 2319339868 0 0) 0x7f5b10012bb0 con 0x7f5b14104210 2024-03-26T04:12:45.622 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:43.761+0000 7f5b067fc700 1 -- 172.21.15.16:0/1565834463 >> v1:172.21.15.16:6800/1828954920 conn(0x7f5b0003a260 legacy=0x7f5b0003c720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T04:12:45.622 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:44.762+0000 7f5b067fc700 1 -- 172.21.15.16:0/1565834463 <== mon.0 v1:172.21.15.16:6789/0 10 ==== mgrmap(e 11) v1 ==== 45800+0+0 (unknown 3314134554 0 0) 0x7f5b100486d0 con 0x7f5b14104210 2024-03-26T04:12:45.622 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:44.762+0000 7f5b067fc700 1 -- 172.21.15.16:0/1565834463 --> v1:172.21.15.16:6800/4042489921 -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5b0003df30 con 0x7f5b0003edc0 2024-03-26T04:12:45.622 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:44.764+0000 7f5b067fc700 1 -- 172.21.15.16:0/1565834463 <== mgr.14152 v1:172.21.15.16:6800/4042489921 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (unknown 0 0 4109177037) 0x7f5b08006e90 con 0x7f5b0003edc0 2024-03-26T04:12:45.622 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:44.770+0000 7f5b1b5fb700 1 -- 172.21.15.16:0/1565834463 --> v1:172.21.15.16:6800/4042489921 -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f5af8002630 con 0x7f5b0003edc0 2024-03-26T04:12:45.622 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:44.770+0000 7f5b067fc700 1 -- 172.21.15.16:0/1565834463 <== mgr.14152 v1:172.21.15.16:6800/4042489921 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (unknown 0 0 3086460295) 0x7f5af8002630 con 0x7f5b0003edc0 2024-03-26T04:12:45.622 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:44.771+0000 7f5b1b5fb700 1 -- 172.21.15.16:0/1565834463 >> v1:172.21.15.16:6800/4042489921 conn(0x7f5b0003edc0 legacy=0x7f5b000411b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:45.622 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:44.771+0000 7f5b1b5fb700 1 -- 172.21.15.16:0/1565834463 >> v1:172.21.15.16:6789/0 conn(0x7f5b14104210 legacy=0x7f5b14106a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:45.622 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:44.771+0000 7f5b1b5fb700 1 -- 172.21.15.16:0/1565834463 shutdown_connections 2024-03-26T04:12:45.623 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:44.771+0000 7f5b1b5fb700 1 -- 172.21.15.16:0/1565834463 >> 172.21.15.16:0/1565834463 conn(0x7f5b140ffb00 msgr2=0x7f5b14101f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:45.623 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:44.771+0000 7f5b1b5fb700 1 -- 172.21.15.16:0/1565834463 shutdown_connections 2024-03-26T04:12:45.623 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:44.771+0000 7f5b1b5fb700 1 -- 172.21.15.16:0/1565834463 wait complete. 2024-03-26T04:12:45.623 INFO:teuthology.orchestra.run.smithi016.stdout:mgr epoch 9 is available 2024-03-26T04:12:45.623 INFO:teuthology.orchestra.run.smithi016.stdout:Generating a dashboard self-signed certificate... 2024-03-26T04:12:45.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:45 smithi016 ceph-mon[29101]: from='client.14156 v1:172.21.15.16:0/1565834463' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-26T04:12:45.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:45 smithi016 ceph-mon[29101]: from='client.14156 v1:172.21.15.16:0/1565834463' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-26T04:12:47.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:46 smithi016 ceph-mon[29101]: mgrmap e12: y(active, since 2s) 2024-03-26T04:12:47.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:46 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:12:47.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:46 smithi016 ceph-mon[29101]: from='client.14164 v1:172.21.15.16:0/698664095' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:12:47.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:46 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:12:47.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:46 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:12:47.175 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-26T04:12:47.175 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:46.300+0000 7f6a43986700 1 Processor -- start 2024-03-26T04:12:47.175 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:46.301+0000 7f6a43986700 1 -- start start 2024-03-26T04:12:47.175 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:46.301+0000 7f6a43986700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6a3c078e80 con 0x7f6a3c1047c0 2024-03-26T04:12:47.175 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:46.301+0000 7f6a41722700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6a3c1047c0 0x7f6a3c104be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60758/0 (socket says 172.21.15.16:60758) 2024-03-26T04:12:47.175 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:46.301+0000 7f6a41722700 1 -- 172.21.15.16:0/2627309162 learned_addr learned my addr 172.21.15.16:0/2627309162 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:47.175 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:46.301+0000 7f6a2ffff700 1 -- 172.21.15.16:0/2627309162 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2068581797 0 0) 0x7f6a3c078e80 con 0x7f6a3c1047c0 2024-03-26T04:12:47.176 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:46.301+0000 7f6a2ffff700 1 -- 172.21.15.16:0/2627309162 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a280035b0 con 0x7f6a3c1047c0 2024-03-26T04:12:47.176 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:46.301+0000 7f6a2ffff700 1 -- 172.21.15.16:0/2627309162 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2875715618 0 0) 0x7f6a280035b0 con 0x7f6a3c1047c0 2024-03-26T04:12:47.176 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:46.301+0000 7f6a2ffff700 1 -- 172.21.15.16:0/2627309162 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a3c106300 con 0x7f6a3c1047c0 2024-03-26T04:12:47.176 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:46.301+0000 7f6a2ffff700 1 -- 172.21.15.16:0/2627309162 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f6a38001580 con 0x7f6a3c1047c0 2024-03-26T04:12:47.176 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:46.302+0000 7f6a2ffff700 1 -- 172.21.15.16:0/2627309162 <== mon.0 v1:172.21.15.16:6789/0 4 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f6a38001d00 con 0x7f6a3c1047c0 2024-03-26T04:12:47.176 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:46.302+0000 7f6a2ffff700 1 -- 172.21.15.16:0/2627309162 <== mon.0 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f6a38005960 con 0x7f6a3c1047c0 2024-03-26T04:12:47.176 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:46.302+0000 7f6a43986700 1 -- 172.21.15.16:0/2627309162 >> v1:172.21.15.16:6789/0 conn(0x7f6a3c1047c0 legacy=0x7f6a3c104be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:47.176 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:46.302+0000 7f6a43986700 1 -- 172.21.15.16:0/2627309162 shutdown_connections 2024-03-26T04:12:47.176 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:46.302+0000 7f6a43986700 1 -- 172.21.15.16:0/2627309162 >> 172.21.15.16:0/2627309162 conn(0x7f6a3c1003c0 msgr2=0x7f6a3c102800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:47.176 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:46.303+0000 7f6a43986700 1 -- 172.21.15.16:0/2627309162 shutdown_connections 2024-03-26T04:12:47.176 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:46.303+0000 7f6a43986700 1 -- 172.21.15.16:0/2627309162 wait complete. 2024-03-26T04:12:47.176 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:46.303+0000 7f6a43986700 1 Processor -- start 2024-03-26T04:12:47.176 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:46.303+0000 7f6a43986700 1 -- start start 2024-03-26T04:12:47.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:46.304+0000 7f6a43986700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6a3c19c170 con 0x7f6a3c1047c0 2024-03-26T04:12:47.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:46.304+0000 7f6a41722700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6a3c1047c0 0x7f6a3c19ba60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60764/0 (socket says 172.21.15.16:60764) 2024-03-26T04:12:47.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:46.304+0000 7f6a41722700 1 -- 172.21.15.16:0/698664095 learned_addr learned my addr 172.21.15.16:0/698664095 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:47.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:46.304+0000 7f6a2e7fc700 1 -- 172.21.15.16:0/698664095 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2590098005 0 0) 0x7f6a3c19c170 con 0x7f6a3c1047c0 2024-03-26T04:12:47.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:46.304+0000 7f6a2e7fc700 1 -- 172.21.15.16:0/698664095 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a280055b0 con 0x7f6a3c1047c0 2024-03-26T04:12:47.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:46.304+0000 7f6a2e7fc700 1 -- 172.21.15.16:0/698664095 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2499307378 0 0) 0x7f6a280055b0 con 0x7f6a3c1047c0 2024-03-26T04:12:47.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:46.304+0000 7f6a2e7fc700 1 -- 172.21.15.16:0/698664095 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a3c19c170 con 0x7f6a3c1047c0 2024-03-26T04:12:47.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:46.304+0000 7f6a2e7fc700 1 -- 172.21.15.16:0/698664095 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f6a38001080 con 0x7f6a3c1047c0 2024-03-26T04:12:47.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:46.304+0000 7f6a2e7fc700 1 -- 172.21.15.16:0/698664095 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 234458305 0 0) 0x7f6a3c19c170 con 0x7f6a3c1047c0 2024-03-26T04:12:47.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:46.304+0000 7f6a2e7fc700 1 -- 172.21.15.16:0/698664095 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a3c19c300 con 0x7f6a3c1047c0 2024-03-26T04:12:47.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:46.304+0000 7f6a43986700 1 -- 172.21.15.16:0/698664095 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a3c19c5b0 con 0x7f6a3c1047c0 2024-03-26T04:12:47.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:46.304+0000 7f6a43986700 1 -- 172.21.15.16:0/698664095 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a3c1a0100 con 0x7f6a3c1047c0 2024-03-26T04:12:47.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:46.305+0000 7f6a2e7fc700 1 -- 172.21.15.16:0/698664095 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f6a38004a60 con 0x7f6a3c1047c0 2024-03-26T04:12:47.177 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:46.305+0000 7f6a2e7fc700 1 -- 172.21.15.16:0/698664095 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f6a38006140 con 0x7f6a3c1047c0 2024-03-26T04:12:47.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:46.305+0000 7f6a43986700 1 -- 172.21.15.16:0/698664095 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a20005150 con 0x7f6a3c1047c0 2024-03-26T04:12:47.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:46.305+0000 7f6a2e7fc700 1 -- 172.21.15.16:0/698664095 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 12) v1 ==== 45906+0+0 (unknown 4039838222 0 0) 0x7f6a38011600 con 0x7f6a3c1047c0 2024-03-26T04:12:47.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:46.306+0000 7f6a2e7fc700 1 -- 172.21.15.16:0/698664095 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (unknown 285047801 0 0) 0x7f6a380472c0 con 0x7f6a3c1047c0 2024-03-26T04:12:47.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:46.311+0000 7f6a2e7fc700 1 -- 172.21.15.16:0/698664095 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f6a38012e70 con 0x7f6a3c1047c0 2024-03-26T04:12:47.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:46.522+0000 7f6a43986700 1 -- 172.21.15.16:0/698664095 --> v1:172.21.15.16:6800/4042489921 -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f6a20000bc0 con 0x7f6a2803a280 2024-03-26T04:12:47.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:46.560+0000 7f6a2e7fc700 1 -- 172.21.15.16:0/698664095 <== mgr.14152 v1:172.21.15.16:6800/4042489921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (unknown 0 0 3317752739) 0x7f6a20000bc0 con 0x7f6a2803a280 2024-03-26T04:12:47.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:46.562+0000 7f6a43986700 1 -- 172.21.15.16:0/698664095 >> v1:172.21.15.16:6800/4042489921 conn(0x7f6a2803a280 legacy=0x7f6a2803c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:47.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:46.562+0000 7f6a43986700 1 -- 172.21.15.16:0/698664095 >> v1:172.21.15.16:6789/0 conn(0x7f6a3c1047c0 legacy=0x7f6a3c19ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:47.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:46.562+0000 7f6a43986700 1 -- 172.21.15.16:0/698664095 shutdown_connections 2024-03-26T04:12:47.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:46.562+0000 7f6a43986700 1 -- 172.21.15.16:0/698664095 >> 172.21.15.16:0/698664095 conn(0x7f6a3c1003c0 msgr2=0x7f6a3c102800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:47.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:46.563+0000 7f6a43986700 1 -- 172.21.15.16:0/698664095 shutdown_connections 2024-03-26T04:12:47.178 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:46.563+0000 7f6a43986700 1 -- 172.21.15.16:0/698664095 wait complete. 2024-03-26T04:12:47.178 INFO:teuthology.orchestra.run.smithi016.stdout:Creating initial admin user... 2024-03-26T04:12:48.375 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:48 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:12:48.375 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:48 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:12:49.448 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$NtgOuH//Rdt4loivIQYxMOvj/ZICHANr42yL.lec4FvTcDoFElnqi", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711426368, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-26T04:12:49.449 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:47.591+0000 7febfc16e700 1 Processor -- start 2024-03-26T04:12:49.449 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:47.591+0000 7febfc16e700 1 -- start start 2024-03-26T04:12:49.449 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:47.592+0000 7febfc16e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7febf410a3b0 con 0x7febf4105f10 2024-03-26T04:12:49.449 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:47.592+0000 7febf9f0a700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7febf4105f10 0x7febf4106350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60766/0 (socket says 172.21.15.16:60766) 2024-03-26T04:12:49.449 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:47.592+0000 7febf9f0a700 1 -- 172.21.15.16:0/665706649 learned_addr learned my addr 172.21.15.16:0/665706649 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:49.449 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:47.592+0000 7febf8f08700 1 -- 172.21.15.16:0/665706649 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2343117807 0 0) 0x7febf410a3b0 con 0x7febf4105f10 2024-03-26T04:12:49.449 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:47.592+0000 7febf8f08700 1 -- 172.21.15.16:0/665706649 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febe00035b0 con 0x7febf4105f10 2024-03-26T04:12:49.449 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:47.592+0000 7febf8f08700 1 -- 172.21.15.16:0/665706649 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1094722463 0 0) 0x7febe00035b0 con 0x7febf4105f10 2024-03-26T04:12:49.449 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:47.592+0000 7febf8f08700 1 -- 172.21.15.16:0/665706649 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf410b550 con 0x7febf4105f10 2024-03-26T04:12:49.449 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:47.592+0000 7febf8f08700 1 -- 172.21.15.16:0/665706649 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7febe4001580 con 0x7febf4105f10 2024-03-26T04:12:49.449 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:47.592+0000 7febf8f08700 1 -- 172.21.15.16:0/665706649 <== mon.0 v1:172.21.15.16:6789/0 4 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7febe4001d00 con 0x7febf4105f10 2024-03-26T04:12:49.449 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:47.593+0000 7febf8f08700 1 -- 172.21.15.16:0/665706649 <== mon.0 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7febe4005960 con 0x7febf4105f10 2024-03-26T04:12:49.450 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:47.593+0000 7febfc16e700 1 -- 172.21.15.16:0/665706649 >> v1:172.21.15.16:6789/0 conn(0x7febf4105f10 legacy=0x7febf4106350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:49.450 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:47.593+0000 7febfc16e700 1 -- 172.21.15.16:0/665706649 shutdown_connections 2024-03-26T04:12:49.450 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:47.593+0000 7febfc16e700 1 -- 172.21.15.16:0/665706649 >> 172.21.15.16:0/665706649 conn(0x7febf4101540 msgr2=0x7febf4103960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:49.450 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:47.593+0000 7febfc16e700 1 -- 172.21.15.16:0/665706649 shutdown_connections 2024-03-26T04:12:49.450 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:47.593+0000 7febfc16e700 1 -- 172.21.15.16:0/665706649 wait complete. 2024-03-26T04:12:49.450 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:47.594+0000 7febfc16e700 1 Processor -- start 2024-03-26T04:12:49.450 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:47.594+0000 7febfc16e700 1 -- start start 2024-03-26T04:12:49.450 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:47.594+0000 7febfc16e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7febf419c1b0 con 0x7febf4105f10 2024-03-26T04:12:49.450 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:47.594+0000 7febf9f0a700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7febf4105f10 0x7febf419baa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60772/0 (socket says 172.21.15.16:60772) 2024-03-26T04:12:49.450 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:47.594+0000 7febf9f0a700 1 -- 172.21.15.16:0/3826548225 learned_addr learned my addr 172.21.15.16:0/3826548225 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:49.450 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:47.594+0000 7febeaffd700 1 -- 172.21.15.16:0/3826548225 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2255238789 0 0) 0x7febf419c1b0 con 0x7febf4105f10 2024-03-26T04:12:49.450 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:47.594+0000 7febeaffd700 1 -- 172.21.15.16:0/3826548225 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febe00055b0 con 0x7febf4105f10 2024-03-26T04:12:49.450 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:47.595+0000 7febeaffd700 1 -- 172.21.15.16:0/3826548225 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4077208080 0 0) 0x7febe00055b0 con 0x7febf4105f10 2024-03-26T04:12:49.451 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:47.595+0000 7febeaffd700 1 -- 172.21.15.16:0/3826548225 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febf419c1b0 con 0x7febf4105f10 2024-03-26T04:12:49.451 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:47.595+0000 7febeaffd700 1 -- 172.21.15.16:0/3826548225 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7febe4001080 con 0x7febf4105f10 2024-03-26T04:12:49.451 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:47.595+0000 7febeaffd700 1 -- 172.21.15.16:0/3826548225 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3755655846 0 0) 0x7febf419c1b0 con 0x7febf4105f10 2024-03-26T04:12:49.451 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:47.595+0000 7febeaffd700 1 -- 172.21.15.16:0/3826548225 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf419c340 con 0x7febf4105f10 2024-03-26T04:12:49.451 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:47.595+0000 7febfc16e700 1 -- 172.21.15.16:0/3826548225 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febf419c5f0 con 0x7febf4105f10 2024-03-26T04:12:49.451 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:47.595+0000 7febfc16e700 1 -- 172.21.15.16:0/3826548225 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7febf41a0140 con 0x7febf4105f10 2024-03-26T04:12:49.451 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:47.595+0000 7febeaffd700 1 -- 172.21.15.16:0/3826548225 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7febe4004a60 con 0x7febf4105f10 2024-03-26T04:12:49.451 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:47.595+0000 7febeaffd700 1 -- 172.21.15.16:0/3826548225 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7febe40060d0 con 0x7febf4105f10 2024-03-26T04:12:49.451 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:47.596+0000 7febeaffd700 1 -- 172.21.15.16:0/3826548225 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 12) v1 ==== 45906+0+0 (unknown 4039838222 0 0) 0x7febe40115b0 con 0x7febf4105f10 2024-03-26T04:12:49.451 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:47.596+0000 7febfc16e700 1 -- 172.21.15.16:0/3826548225 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febd8005150 con 0x7febf4105f10 2024-03-26T04:12:49.451 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:47.596+0000 7febeaffd700 1 -- 172.21.15.16:0/3826548225 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (unknown 285047801 0 0) 0x7febe40472e0 con 0x7febf4105f10 2024-03-26T04:12:49.452 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:47.601+0000 7febeaffd700 1 -- 172.21.15.16:0/3826548225 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7febe4012e60 con 0x7febf4105f10 2024-03-26T04:12:49.452 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:47.838+0000 7febfc16e700 1 -- 172.21.15.16:0/3826548225 --> v1:172.21.15.16:6800/4042489921 -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7febd8002430 con 0x7febe003a260 2024-03-26T04:12:49.452 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:48.084+0000 7febeaffd700 1 -- 172.21.15.16:0/3826548225 <== mgr.14152 v1:172.21.15.16:6800/4042489921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (unknown 0 0 3319937054) 0x7febd8002430 con 0x7febe003a260 2024-03-26T04:12:49.452 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:48.086+0000 7febfc16e700 1 -- 172.21.15.16:0/3826548225 >> v1:172.21.15.16:6800/4042489921 conn(0x7febe003a260 legacy=0x7febe003c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:49.452 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:48.086+0000 7febfc16e700 1 -- 172.21.15.16:0/3826548225 >> v1:172.21.15.16:6789/0 conn(0x7febf4105f10 legacy=0x7febf419baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:49.452 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:48.087+0000 7febfc16e700 1 -- 172.21.15.16:0/3826548225 shutdown_connections 2024-03-26T04:12:49.452 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:48.087+0000 7febfc16e700 1 -- 172.21.15.16:0/3826548225 >> 172.21.15.16:0/3826548225 conn(0x7febf4101540 msgr2=0x7febf4101920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:49.452 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:48.087+0000 7febfc16e700 1 -- 172.21.15.16:0/3826548225 shutdown_connections 2024-03-26T04:12:49.452 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:48.087+0000 7febfc16e700 1 -- 172.21.15.16:0/3826548225 wait complete. 2024-03-26T04:12:49.452 INFO:teuthology.orchestra.run.smithi016.stdout:Fetching dashboard port number... 2024-03-26T04:12:49.567 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:49 smithi016 ceph-mon[29101]: from='client.14166 v1:172.21.15.16:0/3826548225' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:12:50.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:50 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3694991974' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-26T04:12:50.747 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stdout 8443 2024-03-26T04:12:50.747 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:49.856+0000 7fabc84cd700 1 Processor -- start 2024-03-26T04:12:50.747 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:49.857+0000 7fabc84cd700 1 -- start start 2024-03-26T04:12:50.747 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:49.857+0000 7fabc84cd700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fabc010a340 con 0x7fabc0105f20 2024-03-26T04:12:50.747 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:49.857+0000 7fabc6269700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fabc0105f20 0x7fabc0106340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60786/0 (socket says 172.21.15.16:60786) 2024-03-26T04:12:50.747 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:49.857+0000 7fabc6269700 1 -- 172.21.15.16:0/2172258580 learned_addr learned my addr 172.21.15.16:0/2172258580 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:50.747 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:49.857+0000 7fabc5267700 1 -- 172.21.15.16:0/2172258580 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 68474328 0 0) 0x7fabc010a340 con 0x7fabc0105f20 2024-03-26T04:12:50.747 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:49.857+0000 7fabc5267700 1 -- 172.21.15.16:0/2172258580 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabac0035b0 con 0x7fabc0105f20 2024-03-26T04:12:50.748 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:49.857+0000 7fabc5267700 1 -- 172.21.15.16:0/2172258580 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2094709106 0 0) 0x7fabac0035b0 con 0x7fabc0105f20 2024-03-26T04:12:50.748 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:49.857+0000 7fabc5267700 1 -- 172.21.15.16:0/2172258580 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc010b4e0 con 0x7fabc0105f20 2024-03-26T04:12:50.748 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:49.857+0000 7fabc5267700 1 -- 172.21.15.16:0/2172258580 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7fabbc001580 con 0x7fabc0105f20 2024-03-26T04:12:50.748 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:49.858+0000 7fabc5267700 1 -- 172.21.15.16:0/2172258580 <== mon.0 v1:172.21.15.16:6789/0 4 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7fabbc001d00 con 0x7fabc0105f20 2024-03-26T04:12:50.748 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:49.858+0000 7fabc5267700 1 -- 172.21.15.16:0/2172258580 <== mon.0 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7fabbc005960 con 0x7fabc0105f20 2024-03-26T04:12:50.748 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:49.858+0000 7fabc84cd700 1 -- 172.21.15.16:0/2172258580 >> v1:172.21.15.16:6789/0 conn(0x7fabc0105f20 legacy=0x7fabc0106340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:50.748 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:49.858+0000 7fabc84cd700 1 -- 172.21.15.16:0/2172258580 shutdown_connections 2024-03-26T04:12:50.748 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:49.858+0000 7fabc84cd700 1 -- 172.21.15.16:0/2172258580 >> 172.21.15.16:0/2172258580 conn(0x7fabc01014e0 msgr2=0x7fabc0103900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:50.748 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:49.858+0000 7fabc84cd700 1 -- 172.21.15.16:0/2172258580 shutdown_connections 2024-03-26T04:12:50.748 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:49.858+0000 7fabc84cd700 1 -- 172.21.15.16:0/2172258580 wait complete. 2024-03-26T04:12:50.748 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:49.859+0000 7fabc84cd700 1 Processor -- start 2024-03-26T04:12:50.748 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:49.859+0000 7fabc84cd700 1 -- start start 2024-03-26T04:12:50.748 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:49.859+0000 7fabc84cd700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fabc0199ec0 con 0x7fabc0105f20 2024-03-26T04:12:50.748 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:49.859+0000 7fabc6269700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fabc0105f20 0x7fabc01997b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60794/0 (socket says 172.21.15.16:60794) 2024-03-26T04:12:50.748 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:49.859+0000 7fabc6269700 1 -- 172.21.15.16:0/3694991974 learned_addr learned my addr 172.21.15.16:0/3694991974 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:50.749 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:49.859+0000 7fabb37fe700 1 -- 172.21.15.16:0/3694991974 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 480098446 0 0) 0x7fabc0199ec0 con 0x7fabc0105f20 2024-03-26T04:12:50.749 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:49.859+0000 7fabb37fe700 1 -- 172.21.15.16:0/3694991974 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabac0055b0 con 0x7fabc0105f20 2024-03-26T04:12:50.749 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:49.859+0000 7fabb37fe700 1 -- 172.21.15.16:0/3694991974 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2469874203 0 0) 0x7fabac0055b0 con 0x7fabc0105f20 2024-03-26T04:12:50.749 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:49.859+0000 7fabb37fe700 1 -- 172.21.15.16:0/3694991974 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fabc0199ec0 con 0x7fabc0105f20 2024-03-26T04:12:50.749 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:49.859+0000 7fabb37fe700 1 -- 172.21.15.16:0/3694991974 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7fabbc001080 con 0x7fabc0105f20 2024-03-26T04:12:50.749 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:49.860+0000 7fabb37fe700 1 -- 172.21.15.16:0/3694991974 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3161959466 0 0) 0x7fabc0199ec0 con 0x7fabc0105f20 2024-03-26T04:12:50.749 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:49.860+0000 7fabb37fe700 1 -- 172.21.15.16:0/3694991974 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc019a050 con 0x7fabc0105f20 2024-03-26T04:12:50.749 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:49.860+0000 7fabc84cd700 1 -- 172.21.15.16:0/3694991974 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabc019a300 con 0x7fabc0105f20 2024-03-26T04:12:50.749 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:49.860+0000 7fabc84cd700 1 -- 172.21.15.16:0/3694991974 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fabc019de50 con 0x7fabc0105f20 2024-03-26T04:12:50.749 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:49.860+0000 7fabb37fe700 1 -- 172.21.15.16:0/3694991974 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7fabbc004a60 con 0x7fabc0105f20 2024-03-26T04:12:50.749 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:49.860+0000 7fabb37fe700 1 -- 172.21.15.16:0/3694991974 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7fabbc006190 con 0x7fabc0105f20 2024-03-26T04:12:50.750 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:49.861+0000 7fabb37fe700 1 -- 172.21.15.16:0/3694991974 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 12) v1 ==== 45906+0+0 (unknown 4039838222 0 0) 0x7fabbc0116f0 con 0x7fabc0105f20 2024-03-26T04:12:50.750 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:49.861+0000 7fabc84cd700 1 -- 172.21.15.16:0/3694991974 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faba4005150 con 0x7fabc0105f20 2024-03-26T04:12:50.750 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:49.861+0000 7fabb37fe700 1 -- 172.21.15.16:0/3694991974 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (unknown 285047801 0 0) 0x7fabbc047650 con 0x7fabc0105f20 2024-03-26T04:12:50.750 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:49.866+0000 7fabb37fe700 1 -- 172.21.15.16:0/3694991974 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fabbc012fb0 con 0x7fabc0105f20 2024-03-26T04:12:50.750 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:50.068+0000 7fabc84cd700 1 -- 172.21.15.16:0/3694991974 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7faba4005980 con 0x7fabc0105f20 2024-03-26T04:12:50.750 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:50.069+0000 7fabb37fe700 1 -- 172.21.15.16:0/3694991974 <== mon.0 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (unknown 3713421687 0 83753974) 0x7fabbc015090 con 0x7fabc0105f20 2024-03-26T04:12:50.750 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:50.071+0000 7fabc84cd700 1 -- 172.21.15.16:0/3694991974 >> v1:172.21.15.16:6800/4042489921 conn(0x7fabac03a230 legacy=0x7fabac03c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:50.750 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:50.071+0000 7fabc84cd700 1 -- 172.21.15.16:0/3694991974 >> v1:172.21.15.16:6789/0 conn(0x7fabc0105f20 legacy=0x7fabc01997b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:50.750 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:50.071+0000 7fabc84cd700 1 -- 172.21.15.16:0/3694991974 shutdown_connections 2024-03-26T04:12:50.750 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:50.071+0000 7fabc84cd700 1 -- 172.21.15.16:0/3694991974 >> 172.21.15.16:0/3694991974 conn(0x7fabc01014e0 msgr2=0x7fabc01038d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:50.750 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:50.071+0000 7fabc84cd700 1 -- 172.21.15.16:0/3694991974 shutdown_connections 2024-03-26T04:12:50.750 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:50.071+0000 7fabc84cd700 1 -- 172.21.15.16:0/3694991974 wait complete. 2024-03-26T04:12:50.756 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-26T04:12:50.756 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout disabled 2024-03-26T04:12:50.763 INFO:teuthology.orchestra.run.smithi016.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-26T04:12:50.764 INFO:teuthology.orchestra.run.smithi016.stdout:systemctl: stdout inactive 2024-03-26T04:12:50.764 INFO:teuthology.orchestra.run.smithi016.stdout:firewalld.service is not enabled 2024-03-26T04:12:50.764 INFO:teuthology.orchestra.run.smithi016.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-26T04:12:50.766 INFO:teuthology.orchestra.run.smithi016.stdout:Ceph Dashboard is now available at: 2024-03-26T04:12:50.766 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:12:50.766 INFO:teuthology.orchestra.run.smithi016.stdout: URL: https://smithi016.front.sepia.ceph.com:8443/ 2024-03-26T04:12:50.766 INFO:teuthology.orchestra.run.smithi016.stdout: User: admin 2024-03-26T04:12:50.766 INFO:teuthology.orchestra.run.smithi016.stdout: Password: a0x2xwvwbo 2024-03-26T04:12:50.766 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:12:50.766 INFO:teuthology.orchestra.run.smithi016.stdout:Saving cluster configuration to /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/config directory 2024-03-26T04:12:50.767 INFO:teuthology.orchestra.run.smithi016.stdout:Enabling autotune for osd_memory_target 2024-03-26T04:12:51.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:51 smithi016 ceph-mon[29101]: mgrmap e13: y(active, since 6s) 2024-03-26T04:12:51.945 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:51.205+0000 7fbd6a1cd700 1 Processor -- start 2024-03-26T04:12:51.945 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:51.205+0000 7fbd6a1cd700 1 -- start start 2024-03-26T04:12:51.945 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:51.205+0000 7fbd6a1cd700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbd64109240 con 0x7fbd64104e20 2024-03-26T04:12:51.945 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:51.205+0000 7fbd637fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbd64104e20 0x7fbd64105240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60808/0 (socket says 172.21.15.16:60808) 2024-03-26T04:12:51.945 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:51.206+0000 7fbd637fe700 1 -- 172.21.15.16:0/3326102907 learned_addr learned my addr 172.21.15.16:0/3326102907 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:51.945 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:51.206+0000 7fbd627fc700 1 -- 172.21.15.16:0/3326102907 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4098281431 0 0) 0x7fbd64109240 con 0x7fbd64104e20 2024-03-26T04:12:51.946 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:51.206+0000 7fbd627fc700 1 -- 172.21.15.16:0/3326102907 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd4c0057c0 con 0x7fbd64104e20 2024-03-26T04:12:51.946 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:51.206+0000 7fbd627fc700 1 -- 172.21.15.16:0/3326102907 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3156509804 0 0) 0x7fbd4c0057c0 con 0x7fbd64104e20 2024-03-26T04:12:51.946 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:51.206+0000 7fbd627fc700 1 -- 172.21.15.16:0/3326102907 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd6410a3e0 con 0x7fbd64104e20 2024-03-26T04:12:51.946 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:51.206+0000 7fbd627fc700 1 -- 172.21.15.16:0/3326102907 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7fbd54001580 con 0x7fbd64104e20 2024-03-26T04:12:51.946 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:51.206+0000 7fbd627fc700 1 -- 172.21.15.16:0/3326102907 <== mon.0 v1:172.21.15.16:6789/0 4 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7fbd54001d00 con 0x7fbd64104e20 2024-03-26T04:12:51.946 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:51.206+0000 7fbd627fc700 1 -- 172.21.15.16:0/3326102907 <== mon.0 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7fbd54005960 con 0x7fbd64104e20 2024-03-26T04:12:51.946 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:51.207+0000 7fbd6a1cd700 1 -- 172.21.15.16:0/3326102907 >> v1:172.21.15.16:6789/0 conn(0x7fbd64104e20 legacy=0x7fbd64105240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:51.946 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:51.207+0000 7fbd6a1cd700 1 -- 172.21.15.16:0/3326102907 shutdown_connections 2024-03-26T04:12:51.946 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:51.207+0000 7fbd6a1cd700 1 -- 172.21.15.16:0/3326102907 >> 172.21.15.16:0/3326102907 conn(0x7fbd641003e0 msgr2=0x7fbd64102800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:51.946 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:51.207+0000 7fbd6a1cd700 1 -- 172.21.15.16:0/3326102907 shutdown_connections 2024-03-26T04:12:51.946 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:51.207+0000 7fbd6a1cd700 1 -- 172.21.15.16:0/3326102907 wait complete. 2024-03-26T04:12:51.946 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:51.207+0000 7fbd6a1cd700 1 Processor -- start 2024-03-26T04:12:51.947 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:51.208+0000 7fbd6a1cd700 1 -- start start 2024-03-26T04:12:51.947 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:51.208+0000 7fbd6a1cd700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbd64197cc0 con 0x7fbd64104e20 2024-03-26T04:12:51.947 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:51.208+0000 7fbd637fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbd64104e20 0x7fbd641975b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60818/0 (socket says 172.21.15.16:60818) 2024-03-26T04:12:51.947 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:51.208+0000 7fbd637fe700 1 -- 172.21.15.16:0/3024375063 learned_addr learned my addr 172.21.15.16:0/3024375063 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:51.947 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:51.208+0000 7fbd60ff9700 1 -- 172.21.15.16:0/3024375063 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 208916819 0 0) 0x7fbd64197cc0 con 0x7fbd64104e20 2024-03-26T04:12:51.947 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:51.208+0000 7fbd60ff9700 1 -- 172.21.15.16:0/3024375063 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd4c0077c0 con 0x7fbd64104e20 2024-03-26T04:12:51.947 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:51.208+0000 7fbd60ff9700 1 -- 172.21.15.16:0/3024375063 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3688051081 0 0) 0x7fbd4c0077c0 con 0x7fbd64104e20 2024-03-26T04:12:51.947 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:51.208+0000 7fbd60ff9700 1 -- 172.21.15.16:0/3024375063 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd64197cc0 con 0x7fbd64104e20 2024-03-26T04:12:51.947 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:51.208+0000 7fbd60ff9700 1 -- 172.21.15.16:0/3024375063 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7fbd54001080 con 0x7fbd64104e20 2024-03-26T04:12:51.947 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:51.209+0000 7fbd60ff9700 1 -- 172.21.15.16:0/3024375063 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3103990477 0 0) 0x7fbd64197cc0 con 0x7fbd64104e20 2024-03-26T04:12:51.947 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:51.209+0000 7fbd60ff9700 1 -- 172.21.15.16:0/3024375063 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd64197e50 con 0x7fbd64104e20 2024-03-26T04:12:51.947 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:51.209+0000 7fbd6a1cd700 1 -- 172.21.15.16:0/3024375063 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd64198100 con 0x7fbd64104e20 2024-03-26T04:12:51.947 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:51.209+0000 7fbd6a1cd700 1 -- 172.21.15.16:0/3024375063 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd6419bc00 con 0x7fbd64104e20 2024-03-26T04:12:51.947 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:51.209+0000 7fbd60ff9700 1 -- 172.21.15.16:0/3024375063 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7fbd540059b0 con 0x7fbd64104e20 2024-03-26T04:12:51.948 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:51.209+0000 7fbd60ff9700 1 -- 172.21.15.16:0/3024375063 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7fbd54006260 con 0x7fbd64104e20 2024-03-26T04:12:51.948 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:51.210+0000 7fbd60ff9700 1 -- 172.21.15.16:0/3024375063 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 13) v1 ==== 45949+0+0 (unknown 1805086598 0 0) 0x7fbd54011810 con 0x7fbd64104e20 2024-03-26T04:12:51.948 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:51.210+0000 7fbd6a1cd700 1 -- 172.21.15.16:0/3024375063 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd50005150 con 0x7fbd64104e20 2024-03-26T04:12:51.948 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:51.210+0000 7fbd60ff9700 1 -- 172.21.15.16:0/3024375063 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (unknown 285047801 0 0) 0x7fbd54047650 con 0x7fbd64104e20 2024-03-26T04:12:51.948 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:51.215+0000 7fbd60ff9700 1 -- 172.21.15.16:0/3024375063 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fbd54013180 con 0x7fbd64104e20 2024-03-26T04:12:51.948 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:51.418+0000 7fbd6a1cd700 1 -- 172.21.15.16:0/3024375063 --> v1:172.21.15.16:6789/0 -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7fbd50005980 con 0x7fbd64104e20 2024-03-26T04:12:51.948 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:51.419+0000 7fbd60ff9700 1 -- 172.21.15.16:0/3024375063 <== mon.0 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (unknown 689008156 0 0) 0x7fbd54015260 con 0x7fbd64104e20 2024-03-26T04:12:51.948 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:51.421+0000 7fbd6a1cd700 1 -- 172.21.15.16:0/3024375063 >> v1:172.21.15.16:6800/4042489921 conn(0x7fbd4c03c490 legacy=0x7fbd4c03e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:51.948 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:51.421+0000 7fbd6a1cd700 1 -- 172.21.15.16:0/3024375063 >> v1:172.21.15.16:6789/0 conn(0x7fbd64104e20 legacy=0x7fbd641975b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:51.948 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:51.421+0000 7fbd6a1cd700 1 -- 172.21.15.16:0/3024375063 shutdown_connections 2024-03-26T04:12:51.948 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:51.421+0000 7fbd6a1cd700 1 -- 172.21.15.16:0/3024375063 >> 172.21.15.16:0/3024375063 conn(0x7fbd641003e0 msgr2=0x7fbd641027d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:51.948 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:51.421+0000 7fbd6a1cd700 1 -- 172.21.15.16:0/3024375063 shutdown_connections 2024-03-26T04:12:51.948 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:51.421+0000 7fbd6a1cd700 1 -- 172.21.15.16:0/3024375063 wait complete. 2024-03-26T04:12:53.160 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:52.340+0000 7f4e8fe38700 1 Processor -- start 2024-03-26T04:12:53.160 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:52.340+0000 7f4e8fe38700 1 -- start start 2024-03-26T04:12:53.160 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:52.340+0000 7f4e8fe38700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4e8810a340 con 0x7f4e88105f20 2024-03-26T04:12:53.160 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:52.340+0000 7f4e8dbd4700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4e88105f20 0x7f4e88106340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60820/0 (socket says 172.21.15.16:60820) 2024-03-26T04:12:53.161 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:52.340+0000 7f4e8dbd4700 1 -- 172.21.15.16:0/212445145 learned_addr learned my addr 172.21.15.16:0/212445145 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:53.161 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:52.341+0000 7f4e8cbd2700 1 -- 172.21.15.16:0/212445145 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3563949166 0 0) 0x7f4e8810a340 con 0x7f4e88105f20 2024-03-26T04:12:53.161 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:52.341+0000 7f4e8cbd2700 1 -- 172.21.15.16:0/212445145 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e740035b0 con 0x7f4e88105f20 2024-03-26T04:12:53.161 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:52.341+0000 7f4e8cbd2700 1 -- 172.21.15.16:0/212445145 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3958851188 0 0) 0x7f4e740035b0 con 0x7f4e88105f20 2024-03-26T04:12:53.161 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:52.341+0000 7f4e8cbd2700 1 -- 172.21.15.16:0/212445145 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e8810b4e0 con 0x7f4e88105f20 2024-03-26T04:12:53.161 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:52.341+0000 7f4e8cbd2700 1 -- 172.21.15.16:0/212445145 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f4e84001580 con 0x7f4e88105f20 2024-03-26T04:12:53.161 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:52.341+0000 7f4e8cbd2700 1 -- 172.21.15.16:0/212445145 <== mon.0 v1:172.21.15.16:6789/0 4 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f4e84001c30 con 0x7f4e88105f20 2024-03-26T04:12:53.161 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:52.341+0000 7f4e8cbd2700 1 -- 172.21.15.16:0/212445145 <== mon.0 v1:172.21.15.16:6789/0 5 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f4e840057e0 con 0x7f4e88105f20 2024-03-26T04:12:53.161 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:52.342+0000 7f4e8fe38700 1 -- 172.21.15.16:0/212445145 >> v1:172.21.15.16:6789/0 conn(0x7f4e88105f20 legacy=0x7f4e88106340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:53.161 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:52.342+0000 7f4e8fe38700 1 -- 172.21.15.16:0/212445145 shutdown_connections 2024-03-26T04:12:53.161 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:52.342+0000 7f4e8fe38700 1 -- 172.21.15.16:0/212445145 >> 172.21.15.16:0/212445145 conn(0x7f4e881014e0 msgr2=0x7f4e88103900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:53.161 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:52.342+0000 7f4e8fe38700 1 -- 172.21.15.16:0/212445145 shutdown_connections 2024-03-26T04:12:53.162 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:52.342+0000 7f4e8fe38700 1 -- 172.21.15.16:0/212445145 wait complete. 2024-03-26T04:12:53.162 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:52.342+0000 7f4e8fe38700 1 Processor -- start 2024-03-26T04:12:53.162 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:52.342+0000 7f4e8fe38700 1 -- start start 2024-03-26T04:12:53.162 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:52.342+0000 7f4e8fe38700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4e88199f00 con 0x7f4e88105f20 2024-03-26T04:12:53.162 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:52.343+0000 7f4e8dbd4700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4e88105f20 0x7f4e881997f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60822/0 (socket says 172.21.15.16:60822) 2024-03-26T04:12:53.162 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:52.343+0000 7f4e8dbd4700 1 -- 172.21.15.16:0/3608722439 learned_addr learned my addr 172.21.15.16:0/3608722439 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:53.162 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:52.343+0000 7f4e7effd700 1 -- 172.21.15.16:0/3608722439 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1527972577 0 0) 0x7f4e88199f00 con 0x7f4e88105f20 2024-03-26T04:12:53.162 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:52.343+0000 7f4e7effd700 1 -- 172.21.15.16:0/3608722439 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e74005600 con 0x7f4e88105f20 2024-03-26T04:12:53.162 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:52.343+0000 7f4e7effd700 1 -- 172.21.15.16:0/3608722439 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 230424503 0 0) 0x7f4e74005600 con 0x7f4e88105f20 2024-03-26T04:12:53.162 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:52.343+0000 7f4e7effd700 1 -- 172.21.15.16:0/3608722439 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e88199f00 con 0x7f4e88105f20 2024-03-26T04:12:53.162 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:52.343+0000 7f4e7effd700 1 -- 172.21.15.16:0/3608722439 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f4e84000ed0 con 0x7f4e88105f20 2024-03-26T04:12:53.162 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:52.343+0000 7f4e7effd700 1 -- 172.21.15.16:0/3608722439 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1780846497 0 0) 0x7f4e88199f00 con 0x7f4e88105f20 2024-03-26T04:12:53.163 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:52.343+0000 7f4e7effd700 1 -- 172.21.15.16:0/3608722439 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e8819a090 con 0x7f4e88105f20 2024-03-26T04:12:53.163 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:52.343+0000 7f4e8fe38700 1 -- 172.21.15.16:0/3608722439 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e8819a340 con 0x7f4e88105f20 2024-03-26T04:12:53.163 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:52.343+0000 7f4e8fe38700 1 -- 172.21.15.16:0/3608722439 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e8819de90 con 0x7f4e88105f20 2024-03-26T04:12:53.163 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:52.343+0000 7f4e7effd700 1 -- 172.21.15.16:0/3608722439 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f4e84001a50 con 0x7f4e88105f20 2024-03-26T04:12:53.163 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:52.343+0000 7f4e7effd700 1 -- 172.21.15.16:0/3608722439 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f4e84006220 con 0x7f4e88105f20 2024-03-26T04:12:53.163 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:52.344+0000 7f4e7effd700 1 -- 172.21.15.16:0/3608722439 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 13) v1 ==== 45949+0+0 (unknown 1805086598 0 0) 0x7f4e84011730 con 0x7f4e88105f20 2024-03-26T04:12:53.163 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:52.344+0000 7f4e8fe38700 1 -- 172.21.15.16:0/3608722439 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e6c005150 con 0x7f4e88105f20 2024-03-26T04:12:53.163 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:52.344+0000 7f4e7effd700 1 -- 172.21.15.16:0/3608722439 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (unknown 285047801 0 0) 0x7f4e84047510 con 0x7f4e88105f20 2024-03-26T04:12:53.163 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:52.349+0000 7f4e7effd700 1 -- 172.21.15.16:0/3608722439 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f4e84005b40 con 0x7f4e88105f20 2024-03-26T04:12:53.163 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:52.663+0000 7f4e8fe38700 1 -- 172.21.15.16:0/3608722439 --> v1:172.21.15.16:6789/0 -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f4e6c005980 con 0x7f4e88105f20 2024-03-26T04:12:53.163 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:52.666+0000 7f4e7effd700 1 -- 172.21.15.16:0/3608722439 <== mon.0 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) v1 ==== 153+0+0 (unknown 38603050 0 0) 0x7f4e840145a0 con 0x7f4e88105f20 2024-03-26T04:12:53.163 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:52.668+0000 7f4e8fe38700 1 -- 172.21.15.16:0/3608722439 >> v1:172.21.15.16:6800/4042489921 conn(0x7f4e7403a2b0 legacy=0x7f4e7403c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:53.163 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:52.668+0000 7f4e8fe38700 1 -- 172.21.15.16:0/3608722439 >> v1:172.21.15.16:6789/0 conn(0x7f4e88105f20 legacy=0x7f4e881997f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:53.164 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:52.668+0000 7f4e8fe38700 1 -- 172.21.15.16:0/3608722439 shutdown_connections 2024-03-26T04:12:53.164 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:52.668+0000 7f4e8fe38700 1 -- 172.21.15.16:0/3608722439 >> 172.21.15.16:0/3608722439 conn(0x7f4e881014e0 msgr2=0x7f4e881038d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:53.164 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:52.668+0000 7f4e8fe38700 1 -- 172.21.15.16:0/3608722439 shutdown_connections 2024-03-26T04:12:53.164 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr 2024-03-26T04:12:52.668+0000 7f4e8fe38700 1 -- 172.21.15.16:0/3608722439 wait complete. 2024-03-26T04:12:53.164 INFO:teuthology.orchestra.run.smithi016.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-26T04:12:53.164 INFO:teuthology.orchestra.run.smithi016.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-26T04:12:53.164 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:12:53.164 INFO:teuthology.orchestra.run.smithi016.stdout: sudo /sbin/cephadm shell --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-26T04:12:53.164 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:12:53.164 INFO:teuthology.orchestra.run.smithi016.stdout:Or, if you are only running a single cluster on this host: 2024-03-26T04:12:53.164 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:12:53.164 INFO:teuthology.orchestra.run.smithi016.stdout: sudo /sbin/cephadm shell 2024-03-26T04:12:53.165 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:12:53.165 INFO:teuthology.orchestra.run.smithi016.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-26T04:12:53.165 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:12:53.165 INFO:teuthology.orchestra.run.smithi016.stdout: ceph telemetry on 2024-03-26T04:12:53.165 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:12:53.165 INFO:teuthology.orchestra.run.smithi016.stdout:For more information see: 2024-03-26T04:12:53.165 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:12:53.165 INFO:teuthology.orchestra.run.smithi016.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-26T04:12:53.165 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:12:53.165 INFO:teuthology.orchestra.run.smithi016.stdout:Bootstrap complete. 2024-03-26T04:12:53.201 INFO:tasks.cephadm:Fetching config... 2024-03-26T04:12:53.202 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-26T04:12:53.202 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-26T04:12:53.217 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-26T04:12:53.217 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-26T04:12:53.217 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-26T04:12:53.271 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-26T04:12:53.272 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-26T04:12:53.272 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-26T04:12:53.339 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-26T04:12:53.339 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-26T04:12:53.339 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-26T04:12:53.393 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-26T04:12:53.393 DEBUG:teuthology.orchestra.run.smithi016:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrn/eVhSc8Fpb1LKkfulVBTPWyeVKYtY2RYklcoMZHwl+T2OiaQ+STPF38Dtiv9cuOCMI2vutwsXr/T45TlOJ2+jcH20xP/9V+qAHiDQI2bypE5XA/CJPZzRVt8yuVKbXD9xQ0sVgNVTTxA8enD3Nn0CtMHjCYGehj/q3M5oTGzwIDjl6e7RX0Z2MHVBHUOZrMo9Lip90kScMeqhpNUkktfdFbLRTc0deiTSDSoJiCDhfBGayZ5TT9eFUSDogsH+1rQ6cU1bJGkJ6f/6smveDDxfxPoHXiZD4CEy/3i8cGXzmyyK/mX/swYnLclng6rvjQ5TDJ2ywP7wgyA519hSfdnnoq255MifeMK9JdXCoBJDpQkr/rTUBj3gzRm6tYidI6aK84MV6VGPSnckJ0oo1QCmp4+uEibW9nbDT6Ak573NT1QzDR/xhmXGUja3YyGyEpsLxidAhcgIbc3qMhpjLkutqM6MOt93hLYNbOcjX6puuxrGyPBMIHe2YkGmg8738= ceph-d7d59826-eb26-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-26T04:12:53.470 INFO:teuthology.orchestra.run.smithi016.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrn/eVhSc8Fpb1LKkfulVBTPWyeVKYtY2RYklcoMZHwl+T2OiaQ+STPF38Dtiv9cuOCMI2vutwsXr/T45TlOJ2+jcH20xP/9V+qAHiDQI2bypE5XA/CJPZzRVt8yuVKbXD9xQ0sVgNVTTxA8enD3Nn0CtMHjCYGehj/q3M5oTGzwIDjl6e7RX0Z2MHVBHUOZrMo9Lip90kScMeqhpNUkktfdFbLRTc0deiTSDSoJiCDhfBGayZ5TT9eFUSDogsH+1rQ6cU1bJGkJ6f/6smveDDxfxPoHXiZD4CEy/3i8cGXzmyyK/mX/swYnLclng6rvjQ5TDJ2ywP7wgyA519hSfdnnoq255MifeMK9JdXCoBJDpQkr/rTUBj3gzRm6tYidI6aK84MV6VGPSnckJ0oo1QCmp4+uEibW9nbDT6Ak573NT1QzDR/xhmXGUja3YyGyEpsLxidAhcgIbc3qMhpjLkutqM6MOt93hLYNbOcjX6puuxrGyPBMIHe2YkGmg8738= ceph-d7d59826-eb26-11ee-95d0-87774f69a715 2024-03-26T04:12:53.486 DEBUG:teuthology.orchestra.run.smithi183:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrn/eVhSc8Fpb1LKkfulVBTPWyeVKYtY2RYklcoMZHwl+T2OiaQ+STPF38Dtiv9cuOCMI2vutwsXr/T45TlOJ2+jcH20xP/9V+qAHiDQI2bypE5XA/CJPZzRVt8yuVKbXD9xQ0sVgNVTTxA8enD3Nn0CtMHjCYGehj/q3M5oTGzwIDjl6e7RX0Z2MHVBHUOZrMo9Lip90kScMeqhpNUkktfdFbLRTc0deiTSDSoJiCDhfBGayZ5TT9eFUSDogsH+1rQ6cU1bJGkJ6f/6smveDDxfxPoHXiZD4CEy/3i8cGXzmyyK/mX/swYnLclng6rvjQ5TDJ2ywP7wgyA519hSfdnnoq255MifeMK9JdXCoBJDpQkr/rTUBj3gzRm6tYidI6aK84MV6VGPSnckJ0oo1QCmp4+uEibW9nbDT6Ak573NT1QzDR/xhmXGUja3YyGyEpsLxidAhcgIbc3qMhpjLkutqM6MOt93hLYNbOcjX6puuxrGyPBMIHe2YkGmg8738= ceph-d7d59826-eb26-11ee-95d0-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-26T04:12:53.532 INFO:teuthology.orchestra.run.smithi183.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrn/eVhSc8Fpb1LKkfulVBTPWyeVKYtY2RYklcoMZHwl+T2OiaQ+STPF38Dtiv9cuOCMI2vutwsXr/T45TlOJ2+jcH20xP/9V+qAHiDQI2bypE5XA/CJPZzRVt8yuVKbXD9xQ0sVgNVTTxA8enD3Nn0CtMHjCYGehj/q3M5oTGzwIDjl6e7RX0Z2MHVBHUOZrMo9Lip90kScMeqhpNUkktfdFbLRTc0deiTSDSoJiCDhfBGayZ5TT9eFUSDogsH+1rQ6cU1bJGkJ6f/6smveDDxfxPoHXiZD4CEy/3i8cGXzmyyK/mX/swYnLclng6rvjQ5TDJ2ywP7wgyA519hSfdnnoq255MifeMK9JdXCoBJDpQkr/rTUBj3gzRm6tYidI6aK84MV6VGPSnckJ0oo1QCmp4+uEibW9nbDT6Ak573NT1QzDR/xhmXGUja3YyGyEpsLxidAhcgIbc3qMhpjLkutqM6MOt93hLYNbOcjX6puuxrGyPBMIHe2YkGmg8738= ceph-d7d59826-eb26-11ee-95d0-87774f69a715 2024-03-26T04:12:53.547 DEBUG:teuthology.orchestra.run.smithi016:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-26T04:12:53.785 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:12:53.906 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:53 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3608722439' entity='client.admin' 2024-03-26T04:12:55.943 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:55.941+0000 7f2984856700 1 -- 172.21.15.16:0/1645854743 >> v1:172.21.15.16:6789/0 conn(0x7f29801008d0 legacy=0x7f2980100cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:55.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:55.941+0000 7f2984856700 1 -- 172.21.15.16:0/1645854743 shutdown_connections 2024-03-26T04:12:55.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:55.941+0000 7f2984856700 1 -- 172.21.15.16:0/1645854743 >> 172.21.15.16:0/1645854743 conn(0x7f29800fbe70 msgr2=0x7f29800fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:55.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:55.941+0000 7f2984856700 1 -- 172.21.15.16:0/1645854743 shutdown_connections 2024-03-26T04:12:55.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:55.941+0000 7f2984856700 1 -- 172.21.15.16:0/1645854743 wait complete. 2024-03-26T04:12:55.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:55.942+0000 7f2984856700 1 Processor -- start 2024-03-26T04:12:55.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:55.942+0000 7f2984856700 1 -- start start 2024-03-26T04:12:55.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:55.942+0000 7f2984856700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2980195840 con 0x7f29801008d0 2024-03-26T04:12:55.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:55.942+0000 7f297e59c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f29801008d0 0x7f2980195130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58674/0 (socket says 172.21.15.16:58674) 2024-03-26T04:12:55.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:55.942+0000 7f297e59c700 1 -- 172.21.15.16:0/3848857852 learned_addr learned my addr 172.21.15.16:0/3848857852 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:55.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:55.942+0000 7f29777fe700 1 -- 172.21.15.16:0/3848857852 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 460295993 0 0) 0x7f2980195840 con 0x7f29801008d0 2024-03-26T04:12:55.944 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:55.942+0000 7f29777fe700 1 -- 172.21.15.16:0/3848857852 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f296c0054f0 con 0x7f29801008d0 2024-03-26T04:12:55.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:55.943+0000 7f29777fe700 1 -- 172.21.15.16:0/3848857852 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2282246733 0 0) 0x7f296c0054f0 con 0x7f29801008d0 2024-03-26T04:12:55.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:55.943+0000 7f29777fe700 1 -- 172.21.15.16:0/3848857852 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2980195840 con 0x7f29801008d0 2024-03-26T04:12:55.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:55.943+0000 7f29777fe700 1 -- 172.21.15.16:0/3848857852 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f2978004440 con 0x7f29801008d0 2024-03-26T04:12:55.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:55.943+0000 7f29777fe700 1 -- 172.21.15.16:0/3848857852 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3215050941 0 0) 0x7f2980195840 con 0x7f29801008d0 2024-03-26T04:12:55.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:55.943+0000 7f29777fe700 1 -- 172.21.15.16:0/3848857852 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29801959d0 con 0x7f29801008d0 2024-03-26T04:12:55.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:55.943+0000 7f2984856700 1 -- 172.21.15.16:0/3848857852 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2980195c80 con 0x7f29801008d0 2024-03-26T04:12:55.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:55.943+0000 7f2984856700 1 -- 172.21.15.16:0/3848857852 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f29801997d0 con 0x7f29801008d0 2024-03-26T04:12:55.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:55.943+0000 7f29777fe700 1 -- 172.21.15.16:0/3848857852 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f2978001620 con 0x7f29801008d0 2024-03-26T04:12:55.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:55.943+0000 7f29777fe700 1 -- 172.21.15.16:0/3848857852 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f2978004fd0 con 0x7f29801008d0 2024-03-26T04:12:55.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:55.944+0000 7f29777fe700 1 -- 172.21.15.16:0/3848857852 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 13) v1 ==== 45949+0+0 (unknown 1805086598 0 0) 0x7f2978005150 con 0x7f29801008d0 2024-03-26T04:12:55.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:55.944+0000 7f2984856700 1 -- 172.21.15.16:0/3848857852 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29800636b0 con 0x7f29801008d0 2024-03-26T04:12:55.946 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:55.944+0000 7f29777fe700 1 -- 172.21.15.16:0/3848857852 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (unknown 285047801 0 0) 0x7f2978047440 con 0x7f29801008d0 2024-03-26T04:12:55.950 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:55.949+0000 7f29777fe700 1 -- 172.21.15.16:0/3848857852 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f2978013400 con 0x7f29801008d0 2024-03-26T04:12:56.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:56.150+0000 7f2984856700 1 -- 172.21.15.16:0/3848857852 --> v1:172.21.15.16:6789/0 -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f2980106f00 con 0x7f29801008d0 2024-03-26T04:12:56.155 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:56.153+0000 7f29777fe700 1 -- 172.21.15.16:0/3848857852 <== mon.0 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (unknown 3028693289 0 0) 0x7f2978014f50 con 0x7f29801008d0 2024-03-26T04:12:56.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:56.156+0000 7f2984856700 1 -- 172.21.15.16:0/3848857852 >> v1:172.21.15.16:6800/4042489921 conn(0x7f296c03a240 legacy=0x7f296c03c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:56.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:56.156+0000 7f2984856700 1 -- 172.21.15.16:0/3848857852 >> v1:172.21.15.16:6789/0 conn(0x7f29801008d0 legacy=0x7f2980195130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:56.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:56.156+0000 7f2984856700 1 -- 172.21.15.16:0/3848857852 shutdown_connections 2024-03-26T04:12:56.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:56.156+0000 7f2984856700 1 -- 172.21.15.16:0/3848857852 >> 172.21.15.16:0/3848857852 conn(0x7f29800fbe70 msgr2=0x7f29800fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:56.157 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:56.156+0000 7f2984856700 1 -- 172.21.15.16:0/3848857852 shutdown_connections 2024-03-26T04:12:56.158 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:56.156+0000 7f2984856700 1 -- 172.21.15.16:0/3848857852 wait complete. 2024-03-26T04:12:56.169 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:56 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:12:56.169 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:56 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:12:56.169 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:56 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi016", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:12:56.169 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:56 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:12:56.169 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:56 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:12:56.169 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:56 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:12:56.752 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-26T04:12:56.752 DEBUG:teuthology.orchestra.run.smithi016:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-26T04:12:56.992 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:12:57.393 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:57 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3848857852' entity='client.admin' 2024-03-26T04:12:57.394 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:57 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:12:57.394 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:57 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:12:58.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:58.392+0000 7f42c93dd700 1 -- 172.21.15.16:0/2810202928 >> v1:172.21.15.16:6789/0 conn(0x7f42c4100910 legacy=0x7f42c4100d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:58.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:58.393+0000 7f42c93dd700 1 -- 172.21.15.16:0/2810202928 shutdown_connections 2024-03-26T04:12:58.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:58.393+0000 7f42c93dd700 1 -- 172.21.15.16:0/2810202928 >> 172.21.15.16:0/2810202928 conn(0x7f42c40fbeb0 msgr2=0x7f42c40fe2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:58.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:58.393+0000 7f42c93dd700 1 -- 172.21.15.16:0/2810202928 shutdown_connections 2024-03-26T04:12:58.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:58.393+0000 7f42c93dd700 1 -- 172.21.15.16:0/2810202928 wait complete. 2024-03-26T04:12:58.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:58.393+0000 7f42c93dd700 1 Processor -- start 2024-03-26T04:12:58.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:58.393+0000 7f42c93dd700 1 -- start start 2024-03-26T04:12:58.395 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:58.394+0000 7f42c93dd700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f42c4195860 con 0x7f42c4100910 2024-03-26T04:12:58.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:58.394+0000 7f42c2ffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f42c4100910 0x7f42c4195150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58684/0 (socket says 172.21.15.16:58684) 2024-03-26T04:12:58.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:58.394+0000 7f42c2ffd700 1 -- 172.21.15.16:0/1621738011 learned_addr learned my addr 172.21.15.16:0/1621738011 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:12:58.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:58.394+0000 7f42bbfff700 1 -- 172.21.15.16:0/1621738011 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 590913162 0 0) 0x7f42c4195860 con 0x7f42c4100910 2024-03-26T04:12:58.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:58.394+0000 7f42bbfff700 1 -- 172.21.15.16:0/1621738011 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42b00054f0 con 0x7f42c4100910 2024-03-26T04:12:58.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:58.394+0000 7f42bbfff700 1 -- 172.21.15.16:0/1621738011 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3234250465 0 0) 0x7f42b00054f0 con 0x7f42c4100910 2024-03-26T04:12:58.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:58.394+0000 7f42bbfff700 1 -- 172.21.15.16:0/1621738011 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42c4195860 con 0x7f42c4100910 2024-03-26T04:12:58.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:58.394+0000 7f42bbfff700 1 -- 172.21.15.16:0/1621738011 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f42b40047c0 con 0x7f42c4100910 2024-03-26T04:12:58.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:58.394+0000 7f42bbfff700 1 -- 172.21.15.16:0/1621738011 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2069305407 0 0) 0x7f42c4195860 con 0x7f42c4100910 2024-03-26T04:12:58.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:58.394+0000 7f42bbfff700 1 -- 172.21.15.16:0/1621738011 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42c41959f0 con 0x7f42c4100910 2024-03-26T04:12:58.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:58.394+0000 7f42c93dd700 1 -- 172.21.15.16:0/1621738011 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42c4195ca0 con 0x7f42c4100910 2024-03-26T04:12:58.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:58.394+0000 7f42c93dd700 1 -- 172.21.15.16:0/1621738011 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f42c41997f0 con 0x7f42c4100910 2024-03-26T04:12:58.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:58.395+0000 7f42bbfff700 1 -- 172.21.15.16:0/1621738011 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f42b4001510 con 0x7f42c4100910 2024-03-26T04:12:58.397 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:58.395+0000 7f42bbfff700 1 -- 172.21.15.16:0/1621738011 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f42b4005140 con 0x7f42c4100910 2024-03-26T04:12:58.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:58.395+0000 7f42bbfff700 1 -- 172.21.15.16:0/1621738011 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 13) v1 ==== 45949+0+0 (unknown 1805086598 0 0) 0x7f42b40052c0 con 0x7f42c4100910 2024-03-26T04:12:58.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:58.395+0000 7f42c93dd700 1 -- 172.21.15.16:0/1621738011 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42a4005150 con 0x7f42c4100910 2024-03-26T04:12:58.398 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:58.396+0000 7f42bbfff700 1 -- 172.21.15.16:0/1621738011 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (unknown 285047801 0 0) 0x7f42b4047660 con 0x7f42c4100910 2024-03-26T04:12:58.401 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:58.400+0000 7f42bbfff700 1 -- 172.21.15.16:0/1621738011 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f42b40056b0 con 0x7f42c4100910 2024-03-26T04:12:58.609 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:58.607+0000 7f42c93dd700 1 -- 172.21.15.16:0/1621738011 --> v1:172.21.15.16:6800/4042489921 -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f42a4000bc0 con 0x7f42b003a290 2024-03-26T04:12:58.612 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:58.611+0000 7f42bbfff700 1 -- 172.21.15.16:0/1621738011 <== mgr.14152 v1:172.21.15.16:6800/4042489921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (unknown 0 0 0) 0x7f42a4000bc0 con 0x7f42b003a290 2024-03-26T04:12:58.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:58.613+0000 7f42c93dd700 1 -- 172.21.15.16:0/1621738011 >> v1:172.21.15.16:6800/4042489921 conn(0x7f42b003a290 legacy=0x7f42b003c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:58.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:58.613+0000 7f42c93dd700 1 -- 172.21.15.16:0/1621738011 >> v1:172.21.15.16:6789/0 conn(0x7f42c4100910 legacy=0x7f42c4195150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:12:58.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:58.613+0000 7f42c93dd700 1 -- 172.21.15.16:0/1621738011 shutdown_connections 2024-03-26T04:12:58.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:58.614+0000 7f42c93dd700 1 -- 172.21.15.16:0/1621738011 >> 172.21.15.16:0/1621738011 conn(0x7f42c40fbeb0 msgr2=0x7f42c40fe2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:12:58.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:58.614+0000 7f42c93dd700 1 -- 172.21.15.16:0/1621738011 shutdown_connections 2024-03-26T04:12:58.615 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:12:58.614+0000 7f42c93dd700 1 -- 172.21.15.16:0/1621738011 wait complete. 2024-03-26T04:12:59.394 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi183 2024-03-26T04:12:59.394 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-26T04:12:59.395 DEBUG:teuthology.orchestra.run.smithi183:> dd of=/etc/ceph/ceph.conf 2024-03-26T04:12:59.414 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-26T04:12:59.415 DEBUG:teuthology.orchestra.run.smithi183:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:12:59.471 INFO:tasks.cephadm:Adding host smithi183 to orchestrator... 2024-03-26T04:12:59.472 DEBUG:teuthology.orchestra.run.smithi016:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph orch host add smithi183 2024-03-26T04:12:59.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:59 smithi016 ceph-mon[29101]: from='client.14176 v1:172.21.15.16:0/1621738011' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:12:59.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:59 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:12:59.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:59 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:12:59.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:59 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:12:59.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:59 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:12:59.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:59 smithi016 ceph-mon[29101]: Updating smithi016:/etc/ceph/ceph.conf 2024-03-26T04:12:59.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:59 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:12:59.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:59 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:12:59.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:12:59 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:12:59.709 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:13:00.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:00 smithi016 ceph-mon[29101]: Updating smithi016:/var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:13:00.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:00 smithi016 ceph-mon[29101]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:13:00.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:00 smithi016 ceph-mon[29101]: Updating smithi016:/var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:13:01.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:01.247+0000 7fe729fa3700 1 -- 172.21.15.16:0/3331148291 >> v1:172.21.15.16:6789/0 conn(0x7fe724106890 legacy=0x7fe724106c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:13:01.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:01.247+0000 7fe729fa3700 1 -- 172.21.15.16:0/3331148291 shutdown_connections 2024-03-26T04:13:01.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:01.247+0000 7fe729fa3700 1 -- 172.21.15.16:0/3331148291 >> 172.21.15.16:0/3331148291 conn(0x7fe7240fc3a0 msgr2=0x7fe7240fe7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:13:01.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:01.247+0000 7fe729fa3700 1 -- 172.21.15.16:0/3331148291 shutdown_connections 2024-03-26T04:13:01.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:01.247+0000 7fe729fa3700 1 -- 172.21.15.16:0/3331148291 wait complete. 2024-03-26T04:13:01.249 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:01.248+0000 7fe729fa3700 1 Processor -- start 2024-03-26T04:13:01.250 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:01.248+0000 7fe729fa3700 1 -- start start 2024-03-26T04:13:01.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:01.248+0000 7fe729fa3700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe724195b50 con 0x7fe724106890 2024-03-26T04:13:01.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:01.248+0000 7fe7237fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe724106890 0x7fe724195440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58690/0 (socket says 172.21.15.16:58690) 2024-03-26T04:13:01.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:01.248+0000 7fe7237fe700 1 -- 172.21.15.16:0/2934341995 learned_addr learned my addr 172.21.15.16:0/2934341995 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:13:01.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:01.249+0000 7fe7217fa700 1 -- 172.21.15.16:0/2934341995 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1302255959 0 0) 0x7fe724195b50 con 0x7fe724106890 2024-03-26T04:13:01.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:01.249+0000 7fe7217fa700 1 -- 172.21.15.16:0/2934341995 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7100054f0 con 0x7fe724106890 2024-03-26T04:13:01.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:01.249+0000 7fe7217fa700 1 -- 172.21.15.16:0/2934341995 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 583938448 0 0) 0x7fe7100054f0 con 0x7fe724106890 2024-03-26T04:13:01.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:01.249+0000 7fe7217fa700 1 -- 172.21.15.16:0/2934341995 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe724195b50 con 0x7fe724106890 2024-03-26T04:13:01.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:01.249+0000 7fe7217fa700 1 -- 172.21.15.16:0/2934341995 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7fe714001980 con 0x7fe724106890 2024-03-26T04:13:01.251 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:01.249+0000 7fe7217fa700 1 -- 172.21.15.16:0/2934341995 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 754742442 0 0) 0x7fe724195b50 con 0x7fe724106890 2024-03-26T04:13:01.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:01.249+0000 7fe7217fa700 1 -- 172.21.15.16:0/2934341995 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe724195ce0 con 0x7fe724106890 2024-03-26T04:13:01.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:01.249+0000 7fe729fa3700 1 -- 172.21.15.16:0/2934341995 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe724195f90 con 0x7fe724106890 2024-03-26T04:13:01.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:01.249+0000 7fe729fa3700 1 -- 172.21.15.16:0/2934341995 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe72419e170 con 0x7fe724106890 2024-03-26T04:13:01.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:01.249+0000 7fe7217fa700 1 -- 172.21.15.16:0/2934341995 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7fe714001620 con 0x7fe724106890 2024-03-26T04:13:01.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:01.249+0000 7fe7217fa700 1 -- 172.21.15.16:0/2934341995 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7fe714004dc0 con 0x7fe724106890 2024-03-26T04:13:01.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:01.250+0000 7fe7217fa700 1 -- 172.21.15.16:0/2934341995 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 13) v1 ==== 45949+0+0 (unknown 1805086598 0 0) 0x7fe714004f40 con 0x7fe724106890 2024-03-26T04:13:01.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:01.250+0000 7fe729fa3700 1 -- 172.21.15.16:0/2934341995 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe704005120 con 0x7fe724106890 2024-03-26T04:13:01.252 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:01.250+0000 7fe7217fa700 1 -- 172.21.15.16:0/2934341995 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (unknown 285047801 0 0) 0x7fe714047290 con 0x7fe724106890 2024-03-26T04:13:01.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:01.255+0000 7fe7217fa700 1 -- 172.21.15.16:0/2934341995 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fe7140052e0 con 0x7fe724106890 2024-03-26T04:13:01.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:01.463+0000 7fe729fa3700 1 -- 172.21.15.16:0/2934341995 --> v1:172.21.15.16:6800/4042489921 -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi183", "target": ["mon-mgr", ""]}) v1 -- 0x7fe704000b90 con 0x7fe71003a2f0 2024-03-26T04:13:02.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:01 smithi016 ceph-mon[29101]: from='client.14178 v1:172.21.15.16:0/2934341995' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi183", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:13:03.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:02 smithi016 ceph-mon[29101]: Deploying cephadm binary to smithi183 2024-03-26T04:13:05.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:04 smithi016 ceph-mon[29101]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:05.831 INFO:teuthology.orchestra.run.smithi016.stdout:Added host 'smithi183' with addr '172.21.15.183' 2024-03-26T04:13:05.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:05.829+0000 7fe7217fa700 1 -- 172.21.15.16:0/2934341995 <== mgr.14152 v1:172.21.15.16:6800/4042489921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (unknown 0 0 696227864) 0x7fe704000b90 con 0x7fe71003a2f0 2024-03-26T04:13:05.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:05.832+0000 7fe729fa3700 1 -- 172.21.15.16:0/2934341995 >> v1:172.21.15.16:6800/4042489921 conn(0x7fe71003a2f0 legacy=0x7fe71003c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:13:05.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:05.832+0000 7fe729fa3700 1 -- 172.21.15.16:0/2934341995 >> v1:172.21.15.16:6789/0 conn(0x7fe724106890 legacy=0x7fe724195440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:13:05.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:05.832+0000 7fe729fa3700 1 -- 172.21.15.16:0/2934341995 shutdown_connections 2024-03-26T04:13:05.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:05.832+0000 7fe729fa3700 1 -- 172.21.15.16:0/2934341995 >> 172.21.15.16:0/2934341995 conn(0x7fe7240fc3a0 msgr2=0x7fe7240fc780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:13:05.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:05.832+0000 7fe729fa3700 1 -- 172.21.15.16:0/2934341995 shutdown_connections 2024-03-26T04:13:05.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:05.832+0000 7fe729fa3700 1 -- 172.21.15.16:0/2934341995 wait complete. 2024-03-26T04:13:06.453 DEBUG:teuthology.orchestra.run.smithi016:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph orch host ls --format=json 2024-03-26T04:13:06.697 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:13:07.090 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:06 smithi016 ceph-mon[29101]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:07.090 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:06 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:07.090 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:06 smithi016 ceph-mon[29101]: Added host smithi183 2024-03-26T04:13:07.090 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:06 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:13:07.090 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:06 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:07.943 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:07 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:08.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:08.139+0000 7efd5cf53700 1 -- 172.21.15.16:0/2558932618 >> v1:172.21.15.16:6789/0 conn(0x7efd58106870 legacy=0x7efd58106c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:13:08.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:08.139+0000 7efd5cf53700 1 -- 172.21.15.16:0/2558932618 shutdown_connections 2024-03-26T04:13:08.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:08.139+0000 7efd5cf53700 1 -- 172.21.15.16:0/2558932618 >> 172.21.15.16:0/2558932618 conn(0x7efd580fc380 msgr2=0x7efd580fe7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:13:08.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:08.139+0000 7efd5cf53700 1 -- 172.21.15.16:0/2558932618 shutdown_connections 2024-03-26T04:13:08.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:08.139+0000 7efd5cf53700 1 -- 172.21.15.16:0/2558932618 wait complete. 2024-03-26T04:13:08.141 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:08.140+0000 7efd5cf53700 1 Processor -- start 2024-03-26T04:13:08.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:08.140+0000 7efd5cf53700 1 -- start start 2024-03-26T04:13:08.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:08.140+0000 7efd5cf53700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7efd580fe720 con 0x7efd58106870 2024-03-26T04:13:08.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:08.140+0000 7efd5659c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7efd58106870 0x7efd580fc830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43086/0 (socket says 172.21.15.16:43086) 2024-03-26T04:13:08.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:08.140+0000 7efd5659c700 1 -- 172.21.15.16:0/3695379932 learned_addr learned my addr 172.21.15.16:0/3695379932 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:13:08.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:08.140+0000 7efd477fe700 1 -- 172.21.15.16:0/3695379932 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3026156711 0 0) 0x7efd580fe720 con 0x7efd58106870 2024-03-26T04:13:08.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:08.140+0000 7efd477fe700 1 -- 172.21.15.16:0/3695379932 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd400054f0 con 0x7efd58106870 2024-03-26T04:13:08.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:08.141+0000 7efd477fe700 1 -- 172.21.15.16:0/3695379932 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2462727211 0 0) 0x7efd400054f0 con 0x7efd58106870 2024-03-26T04:13:08.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:08.141+0000 7efd477fe700 1 -- 172.21.15.16:0/3695379932 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efd580fe720 con 0x7efd58106870 2024-03-26T04:13:08.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:08.141+0000 7efd477fe700 1 -- 172.21.15.16:0/3695379932 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7efd4c001980 con 0x7efd58106870 2024-03-26T04:13:08.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:08.141+0000 7efd477fe700 1 -- 172.21.15.16:0/3695379932 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3807909419 0 0) 0x7efd580fe720 con 0x7efd58106870 2024-03-26T04:13:08.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:08.141+0000 7efd477fe700 1 -- 172.21.15.16:0/3695379932 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd580fcf40 con 0x7efd58106870 2024-03-26T04:13:08.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:08.141+0000 7efd5cf53700 1 -- 172.21.15.16:0/3695379932 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd580fd1f0 con 0x7efd58106870 2024-03-26T04:13:08.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:08.141+0000 7efd5cf53700 1 -- 172.21.15.16:0/3695379932 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efd580fd670 con 0x7efd58106870 2024-03-26T04:13:08.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:08.141+0000 7efd477fe700 1 -- 172.21.15.16:0/3695379932 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7efd4c001620 con 0x7efd58106870 2024-03-26T04:13:08.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:08.141+0000 7efd477fe700 1 -- 172.21.15.16:0/3695379932 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7efd4c004e50 con 0x7efd58106870 2024-03-26T04:13:08.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:08.142+0000 7efd477fe700 1 -- 172.21.15.16:0/3695379932 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 13) v1 ==== 45949+0+0 (unknown 1805086598 0 0) 0x7efd4c005090 con 0x7efd58106870 2024-03-26T04:13:08.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:08.142+0000 7efd5cf53700 1 -- 172.21.15.16:0/3695379932 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd38005120 con 0x7efd58106870 2024-03-26T04:13:08.143 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:08.142+0000 7efd477fe700 1 -- 172.21.15.16:0/3695379932 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (unknown 285047801 0 0) 0x7efd4c047340 con 0x7efd58106870 2024-03-26T04:13:08.147 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:08.147+0000 7efd477fe700 1 -- 172.21.15.16:0/3695379932 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7efd4c0168f0 con 0x7efd58106870 2024-03-26T04:13:08.357 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:08.357+0000 7efd5cf53700 1 -- 172.21.15.16:0/3695379932 --> v1:172.21.15.16:6800/4042489921 -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efd38000b90 con 0x7efd4003a240 2024-03-26T04:13:08.358 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:08.358+0000 7efd477fe700 1 -- 172.21.15.16:0/3695379932 <== mgr.14152 v1:172.21.15.16:6800/4042489921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (unknown 0 0 39372938) 0x7efd38000b90 con 0x7efd4003a240 2024-03-26T04:13:08.358 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:13:08.358 INFO:teuthology.orchestra.run.smithi016.stdout:[{"addr": "172.21.15.16", "hostname": "smithi016", "labels": [], "status": ""}, {"addr": "172.21.15.183", "hostname": "smithi183", "labels": [], "status": ""}] 2024-03-26T04:13:08.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:08.360+0000 7efd5cf53700 1 -- 172.21.15.16:0/3695379932 >> v1:172.21.15.16:6800/4042489921 conn(0x7efd4003a240 legacy=0x7efd4003c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:13:08.360 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:08.360+0000 7efd5cf53700 1 -- 172.21.15.16:0/3695379932 >> v1:172.21.15.16:6789/0 conn(0x7efd58106870 legacy=0x7efd580fc830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:13:08.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:08.360+0000 7efd5cf53700 1 -- 172.21.15.16:0/3695379932 shutdown_connections 2024-03-26T04:13:08.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:08.360+0000 7efd5cf53700 1 -- 172.21.15.16:0/3695379932 >> 172.21.15.16:0/3695379932 conn(0x7efd580fc380 msgr2=0x7efd58069f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:13:08.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:08.360+0000 7efd5cf53700 1 -- 172.21.15.16:0/3695379932 shutdown_connections 2024-03-26T04:13:08.361 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:08.360+0000 7efd5cf53700 1 -- 172.21.15.16:0/3695379932 wait complete. 2024-03-26T04:13:08.952 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-26T04:13:08.952 DEBUG:teuthology.orchestra.run.smithi016:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph osd crush tunables default 2024-03-26T04:13:09.187 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:13:09.308 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:08 smithi016 ceph-mon[29101]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:09.308 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:08 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:09.309 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:08 smithi016 ceph-mon[29101]: from='client.14180 v1:172.21.15.16:0/3695379932' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:13:10.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:10.754+0000 7f98b8537700 1 -- 172.21.15.16:0/1060881386 >> v1:172.21.15.16:6789/0 conn(0x7f98b0106870 legacy=0x7f98b0106c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:13:10.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:10.755+0000 7f98b8537700 1 -- 172.21.15.16:0/1060881386 shutdown_connections 2024-03-26T04:13:10.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:10.755+0000 7f98b8537700 1 -- 172.21.15.16:0/1060881386 >> 172.21.15.16:0/1060881386 conn(0x7f98b00fc340 msgr2=0x7f98b00fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:13:10.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:10.755+0000 7f98b8537700 1 -- 172.21.15.16:0/1060881386 shutdown_connections 2024-03-26T04:13:10.755 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:10.755+0000 7f98b8537700 1 -- 172.21.15.16:0/1060881386 wait complete. 2024-03-26T04:13:10.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:10.755+0000 7f98b8537700 1 Processor -- start 2024-03-26T04:13:10.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:10.755+0000 7f98b8537700 1 -- start start 2024-03-26T04:13:10.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:10.756+0000 7f98b8537700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f98b0195ba0 con 0x7f98b0106870 2024-03-26T04:13:10.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:10.756+0000 7f98b62d3700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f98b0106870 0x7f98b0195490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43102/0 (socket says 172.21.15.16:43102) 2024-03-26T04:13:10.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:10.756+0000 7f98b62d3700 1 -- 172.21.15.16:0/1934386500 learned_addr learned my addr 172.21.15.16:0/1934386500 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:13:10.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:10.756+0000 7f98af7fe700 1 -- 172.21.15.16:0/1934386500 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1071494995 0 0) 0x7f98b0195ba0 con 0x7f98b0106870 2024-03-26T04:13:10.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:10.756+0000 7f98af7fe700 1 -- 172.21.15.16:0/1934386500 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98980054f0 con 0x7f98b0106870 2024-03-26T04:13:10.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:10.756+0000 7f98af7fe700 1 -- 172.21.15.16:0/1934386500 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 114520698 0 0) 0x7f98980054f0 con 0x7f98b0106870 2024-03-26T04:13:10.757 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:10.756+0000 7f98af7fe700 1 -- 172.21.15.16:0/1934386500 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98b0195ba0 con 0x7f98b0106870 2024-03-26T04:13:10.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:10.756+0000 7f98af7fe700 1 -- 172.21.15.16:0/1934386500 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f98a0001d00 con 0x7f98b0106870 2024-03-26T04:13:10.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:10.756+0000 7f98af7fe700 1 -- 172.21.15.16:0/1934386500 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3358729773 0 0) 0x7f98b0195ba0 con 0x7f98b0106870 2024-03-26T04:13:10.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:10.756+0000 7f98af7fe700 1 -- 172.21.15.16:0/1934386500 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b0195d30 con 0x7f98b0106870 2024-03-26T04:13:10.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:10.756+0000 7f98b8537700 1 -- 172.21.15.16:0/1934386500 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98b0195fe0 con 0x7f98b0106870 2024-03-26T04:13:10.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:10.756+0000 7f98b8537700 1 -- 172.21.15.16:0/1934386500 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f98b019e1c0 con 0x7f98b0106870 2024-03-26T04:13:10.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:10.756+0000 7f98af7fe700 1 -- 172.21.15.16:0/1934386500 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f98a00019a0 con 0x7f98b0106870 2024-03-26T04:13:10.758 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:10.756+0000 7f98af7fe700 1 -- 172.21.15.16:0/1934386500 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f98a00051d0 con 0x7f98b0106870 2024-03-26T04:13:10.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:10.757+0000 7f98af7fe700 1 -- 172.21.15.16:0/1934386500 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 13) v1 ==== 45949+0+0 (unknown 1805086598 0 0) 0x7f98a0005350 con 0x7f98b0106870 2024-03-26T04:13:10.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:10.757+0000 7f98b8537700 1 -- 172.21.15.16:0/1934386500 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f989c005120 con 0x7f98b0106870 2024-03-26T04:13:10.759 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:10.757+0000 7f98af7fe700 1 -- 172.21.15.16:0/1934386500 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 1069+0+0 (unknown 285047801 0 0) 0x7f98a0047880 con 0x7f98b0106870 2024-03-26T04:13:10.762 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:10.762+0000 7f98af7fe700 1 -- 172.21.15.16:0/1934386500 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f98a00056f0 con 0x7f98b0106870 2024-03-26T04:13:10.972 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:10.972+0000 7f98b8537700 1 -- 172.21.15.16:0/1934386500 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f989c0053d0 con 0x7f98b0106870 2024-03-26T04:13:11.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:10 smithi016 ceph-mon[29101]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:11.917 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:11.917+0000 7f98af7fe700 1 -- 172.21.15.16:0/1934386500 <== mon.0 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (unknown 3126668360 0 0) 0x7f98a00151a0 con 0x7f98b0106870 2024-03-26T04:13:11.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:11.919+0000 7f98b8537700 1 -- 172.21.15.16:0/1934386500 >> v1:172.21.15.16:6800/4042489921 conn(0x7f989803a290 legacy=0x7f989803c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:13:11.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:11.919+0000 7f98b8537700 1 -- 172.21.15.16:0/1934386500 >> v1:172.21.15.16:6789/0 conn(0x7f98b0106870 legacy=0x7f98b0195490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:13:11.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:11.920+0000 7f98b8537700 1 -- 172.21.15.16:0/1934386500 shutdown_connections 2024-03-26T04:13:11.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:11.920+0000 7f98b8537700 1 -- 172.21.15.16:0/1934386500 >> 172.21.15.16:0/1934386500 conn(0x7f98b00fc340 msgr2=0x7f98b00fc720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:13:11.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:11.920+0000 7f98b8537700 1 -- 172.21.15.16:0/1934386500 shutdown_connections 2024-03-26T04:13:11.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:11.920+0000 7f98b8537700 1 -- 172.21.15.16:0/1934386500 wait complete. 2024-03-26T04:13:11.921 INFO:teuthology.orchestra.run.smithi016.stderr:adjusted tunables profile to default 2024-03-26T04:13:12.221 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:11 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1934386500' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-26T04:13:12.222 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:11 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:12.222 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:11 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:12.222 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:11 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:12.222 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:11 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:12.222 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:11 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi183", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:13:12.222 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:11 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:12.222 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:11 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:13:12.222 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:11 smithi016 ceph-mon[29101]: Updating smithi183:/etc/ceph/ceph.conf 2024-03-26T04:13:12.222 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:11 smithi016 ceph-mon[29101]: Updating smithi183:/var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:13:12.548 INFO:tasks.cephadm:Adding mon.a on smithi016 2024-03-26T04:13:12.549 INFO:tasks.cephadm:Adding mon.c on smithi016 2024-03-26T04:13:12.549 INFO:tasks.cephadm:Adding mon.b on smithi183 2024-03-26T04:13:12.549 DEBUG:teuthology.orchestra.run.smithi183:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph orch apply mon '3;smithi016:[v1:172.21.15.16:6789]=a;smithi016:[v1:172.21.15.16:6790]=c;smithi183:[v1:172.21.15.183:6789]=b' 2024-03-26T04:13:12.777 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:13:13.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:12 smithi016 ceph-mon[29101]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:13.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:12 smithi016 ceph-mon[29101]: Updating smithi183:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:13:13.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:12 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1934386500' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-26T04:13:13.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:12 smithi016 ceph-mon[29101]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:13:13.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:12 smithi016 ceph-mon[29101]: Updating smithi183:/var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:13:13.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:12 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:13.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:12 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:13.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:12 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:14.201 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:14.200+0000 7f390bd0a700 1 -- 172.21.15.183:0/2325042061 >> v1:172.21.15.16:6789/0 conn(0x7f3904102c10 legacy=0x7f3904102ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:13:14.201 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:14.200+0000 7f390bd0a700 1 -- 172.21.15.183:0/2325042061 shutdown_connections 2024-03-26T04:13:14.202 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:14.200+0000 7f390bd0a700 1 -- 172.21.15.183:0/2325042061 >> 172.21.15.183:0/2325042061 conn(0x7f39040fc3a0 msgr2=0x7f39040fe7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:13:14.202 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:14.200+0000 7f390bd0a700 1 -- 172.21.15.183:0/2325042061 shutdown_connections 2024-03-26T04:13:14.202 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:14.200+0000 7f390bd0a700 1 -- 172.21.15.183:0/2325042061 wait complete. 2024-03-26T04:13:14.202 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:14.201+0000 7f390bd0a700 1 Processor -- start 2024-03-26T04:13:14.202 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:14.201+0000 7f390bd0a700 1 -- start start 2024-03-26T04:13:14.202 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:14.201+0000 7f390bd0a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3904196dd0 con 0x7f3904102c10 2024-03-26T04:13:14.204 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:14.201+0000 7f3909aa6700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3904102c10 0x7f39041966c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.183:49808/0 (socket says 172.21.15.183:49808) 2024-03-26T04:13:14.204 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:14.201+0000 7f3909aa6700 1 -- 172.21.15.183:0/3681984561 learned_addr learned my addr 172.21.15.183:0/3681984561 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-26T04:13:14.204 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:14.202+0000 7f38faffd700 1 -- 172.21.15.183:0/3681984561 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3840447278 0 0) 0x7f3904196dd0 con 0x7f3904102c10 2024-03-26T04:13:14.204 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:14.202+0000 7f38faffd700 1 -- 172.21.15.183:0/3681984561 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38f00054f0 con 0x7f3904102c10 2024-03-26T04:13:14.204 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:14.202+0000 7f38faffd700 1 -- 172.21.15.183:0/3681984561 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2485224352 0 0) 0x7f38f00054f0 con 0x7f3904102c10 2024-03-26T04:13:14.204 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:14.202+0000 7f38faffd700 1 -- 172.21.15.183:0/3681984561 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3904196dd0 con 0x7f3904102c10 2024-03-26T04:13:14.204 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:14.202+0000 7f38faffd700 1 -- 172.21.15.183:0/3681984561 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f39000044b0 con 0x7f3904102c10 2024-03-26T04:13:14.204 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:14.202+0000 7f38faffd700 1 -- 172.21.15.183:0/3681984561 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 628686398 0 0) 0x7f3904196dd0 con 0x7f3904102c10 2024-03-26T04:13:14.204 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:14.202+0000 7f38faffd700 1 -- 172.21.15.183:0/3681984561 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3904196f60 con 0x7f3904102c10 2024-03-26T04:13:14.204 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:14.202+0000 7f390bd0a700 1 -- 172.21.15.183:0/3681984561 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3904197210 con 0x7f3904102c10 2024-03-26T04:13:14.204 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:14.202+0000 7f390bd0a700 1 -- 172.21.15.183:0/3681984561 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f390419f3f0 con 0x7f3904102c10 2024-03-26T04:13:14.204 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:14.202+0000 7f38faffd700 1 -- 172.21.15.183:0/3681984561 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f3900001620 con 0x7f3904102c10 2024-03-26T04:13:14.204 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:14.202+0000 7f38faffd700 1 -- 172.21.15.183:0/3681984561 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f3900004e30 con 0x7f3904102c10 2024-03-26T04:13:14.205 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:14.203+0000 7f38faffd700 1 -- 172.21.15.183:0/3681984561 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 13) v1 ==== 45949+0+0 (unknown 1805086598 0 0) 0x7f3900005030 con 0x7f3904102c10 2024-03-26T04:13:14.205 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:14.203+0000 7f390bd0a700 1 -- 172.21.15.183:0/3681984561 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38e8005120 con 0x7f3904102c10 2024-03-26T04:13:14.205 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:14.203+0000 7f38faffd700 1 -- 172.21.15.183:0/3681984561 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 1069+0+0 (unknown 21079357 0 0) 0x7f3900047400 con 0x7f3904102c10 2024-03-26T04:13:14.209 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:14.209+0000 7f38faffd700 1 -- 172.21.15.183:0/3681984561 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f3900016970 con 0x7f3904102c10 2024-03-26T04:13:14.416 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:14.415+0000 7f390bd0a700 1 -- 172.21.15.183:0/3681984561 --> v1:172.21.15.16:6800/4042489921 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi016:[v1:172.21.15.16:6789]=a;smithi016:[v1:172.21.15.16:6790]=c;smithi183:[v1:172.21.15.183:6789]=b", "target": ["mon-mgr", ""]}) v1 -- 0x7f38e8000c60 con 0x7f38f003a240 2024-03-26T04:13:14.422 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:14.421+0000 7f38faffd700 1 -- 172.21.15.183:0/3681984561 <== mgr.14152 v1:172.21.15.16:6800/4042489921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (unknown 0 0 3265049985) 0x7f38e8000c60 con 0x7f38f003a240 2024-03-26T04:13:14.422 INFO:teuthology.orchestra.run.smithi183.stdout:Scheduled mon update... 2024-03-26T04:13:14.424 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:14.423+0000 7f390bd0a700 1 -- 172.21.15.183:0/3681984561 >> v1:172.21.15.16:6800/4042489921 conn(0x7f38f003a240 legacy=0x7f38f003c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:13:14.424 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:14.423+0000 7f390bd0a700 1 -- 172.21.15.183:0/3681984561 >> v1:172.21.15.16:6789/0 conn(0x7f3904102c10 legacy=0x7f39041966c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:13:14.424 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:14.423+0000 7f390bd0a700 1 -- 172.21.15.183:0/3681984561 shutdown_connections 2024-03-26T04:13:14.424 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:14.423+0000 7f390bd0a700 1 -- 172.21.15.183:0/3681984561 >> 172.21.15.183:0/3681984561 conn(0x7f39040fc3a0 msgr2=0x7f3904108310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:13:14.424 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:14.423+0000 7f390bd0a700 1 -- 172.21.15.183:0/3681984561 shutdown_connections 2024-03-26T04:13:14.424 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:14.423+0000 7f390bd0a700 1 -- 172.21.15.183:0/3681984561 wait complete. 2024-03-26T04:13:15.064 DEBUG:teuthology.orchestra.run.smithi016:mon.c> sudo journalctl -f -n 0 -u ceph-d7d59826-eb26-11ee-95d0-87774f69a715@mon.c.service 2024-03-26T04:13:15.067 DEBUG:teuthology.orchestra.run.smithi183:mon.b> sudo journalctl -f -n 0 -u ceph-d7d59826-eb26-11ee-95d0-87774f69a715@mon.b.service 2024-03-26T04:13:15.069 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T04:13:15.069 DEBUG:teuthology.orchestra.run.smithi183:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T04:13:15.855 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:13:15.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:15 smithi016 ceph-mon[29101]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:15.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:15 smithi016 ceph-mon[29101]: from='client.14184 v1:172.21.15.183:0/3681984561' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi016:[v1:172.21.15.16:6789]=a;smithi016:[v1:172.21.15.16:6790]=c;smithi183:[v1:172.21.15.183:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:13:15.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:15 smithi016 ceph-mon[29101]: Saving service mon spec with placement smithi016:[v1:172.21.15.16:6789]=a;smithi016:[v1:172.21.15.16:6790]=c;smithi183:[v1:172.21.15.183:6789]=b;count:3 2024-03-26T04:13:15.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:15 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:15.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:15 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:13:15.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:15 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:15.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:15 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:13:15.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:15 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:15.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:15 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:13:15.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:15 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:15.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:15 smithi016 ceph-mon[29101]: Deploying daemon mon.b on smithi183 2024-03-26T04:13:17.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:17 smithi016 ceph-mon[29101]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:18.814 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:18.811+0000 7f6915091700 1 -- 172.21.15.183:0/686517618 >> v1:172.21.15.16:6789/0 conn(0x7f6910100920 legacy=0x7f6910100d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:13:18.814 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:18.811+0000 7f6915091700 1 -- 172.21.15.183:0/686517618 shutdown_connections 2024-03-26T04:13:18.814 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:18.811+0000 7f6915091700 1 -- 172.21.15.183:0/686517618 >> 172.21.15.183:0/686517618 conn(0x7f69100fc360 msgr2=0x7f69100fe780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:13:18.814 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:18.811+0000 7f6915091700 1 -- 172.21.15.183:0/686517618 shutdown_connections 2024-03-26T04:13:18.814 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:18.811+0000 7f6915091700 1 -- 172.21.15.183:0/686517618 wait complete. 2024-03-26T04:13:18.814 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:18.812+0000 7f6915091700 1 Processor -- start 2024-03-26T04:13:18.814 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:18.812+0000 7f6915091700 1 -- start start 2024-03-26T04:13:18.815 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:18.812+0000 7f6915091700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f691019d450 con 0x7f6910100920 2024-03-26T04:13:18.815 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:18.813+0000 7f690ed9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6910100920 0x7f691019cd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.183:58052/0 (socket says 172.21.15.183:58052) 2024-03-26T04:13:18.815 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:18.813+0000 7f690ed9d700 1 -- 172.21.15.183:0/827647761 learned_addr learned my addr 172.21.15.183:0/827647761 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-26T04:13:18.815 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:18.813+0000 7f690cd99700 1 -- 172.21.15.183:0/827647761 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2867695131 0 0) 0x7f691019d450 con 0x7f6910100920 2024-03-26T04:13:18.815 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:18.813+0000 7f690cd99700 1 -- 172.21.15.183:0/827647761 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68fc0054f0 con 0x7f6910100920 2024-03-26T04:13:18.815 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:18.813+0000 7f690cd99700 1 -- 172.21.15.183:0/827647761 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2209033489 0 0) 0x7f68fc0054f0 con 0x7f6910100920 2024-03-26T04:13:18.815 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:18.813+0000 7f690cd99700 1 -- 172.21.15.183:0/827647761 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f691019d450 con 0x7f6910100920 2024-03-26T04:13:18.815 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:18.813+0000 7f690cd99700 1 -- 172.21.15.183:0/827647761 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f69000044b0 con 0x7f6910100920 2024-03-26T04:13:18.815 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:18.813+0000 7f690cd99700 1 -- 172.21.15.183:0/827647761 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3623311121 0 0) 0x7f691019d450 con 0x7f6910100920 2024-03-26T04:13:18.815 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:18.813+0000 7f690cd99700 1 -- 172.21.15.183:0/827647761 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f691019d5e0 con 0x7f6910100920 2024-03-26T04:13:18.815 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:18.813+0000 7f6915091700 1 -- 172.21.15.183:0/827647761 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f691019d890 con 0x7f6910100920 2024-03-26T04:13:18.815 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:18.813+0000 7f6915091700 1 -- 172.21.15.183:0/827647761 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f69101a5a70 con 0x7f6910100920 2024-03-26T04:13:18.815 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:18.813+0000 7f690cd99700 1 -- 172.21.15.183:0/827647761 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f6900001620 con 0x7f6910100920 2024-03-26T04:13:18.816 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:18.813+0000 7f690cd99700 1 -- 172.21.15.183:0/827647761 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 170+0+0 (unknown 811239998 0 0) 0x7f6900004e30 con 0x7f6910100920 2024-03-26T04:13:18.816 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:18.814+0000 7f6915091700 1 -- 172.21.15.183:0/827647761 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68f0005120 con 0x7f6910100920 2024-03-26T04:13:18.816 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:18.814+0000 7f690cd99700 1 -- 172.21.15.183:0/827647761 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 13) v1 ==== 45949+0+0 (unknown 1805086598 0 0) 0x7f6900004fb0 con 0x7f6910100920 2024-03-26T04:13:18.816 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:18.814+0000 7f690cd99700 1 -- 172.21.15.183:0/827647761 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 1069+0+0 (unknown 21079357 0 0) 0x7f6900047470 con 0x7f6910100920 2024-03-26T04:13:18.820 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:18.820+0000 7f690cd99700 1 -- 172.21.15.183:0/827647761 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f69000168a0 con 0x7f6910100920 2024-03-26T04:13:19.120 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:19.119+0000 7f6915091700 1 -- 172.21.15.183:0/827647761 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f68f00053d0 con 0x7f6910100920 2024-03-26T04:13:19.120 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:19.119+0000 7f690cd99700 1 -- 172.21.15.183:0/827647761 <== mon.0 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (unknown 2237029548 0 2457150220) 0x7f6900014ea0 con 0x7f6910100920 2024-03-26T04:13:19.120 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:13:19.120 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":1,"fsid":"d7d59826-eb26-11ee-95d0-87774f69a715","modified":"2024-03-26T04:11:44.857505Z","created":"2024-03-26T04:11:44.857505Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-26T04:13:19.122 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:19.121+0000 7f6915091700 1 -- 172.21.15.183:0/827647761 >> v1:172.21.15.16:6800/4042489921 conn(0x7f68fc03a290 legacy=0x7f68fc03c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:13:19.122 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:19.121+0000 7f6915091700 1 -- 172.21.15.183:0/827647761 >> v1:172.21.15.16:6789/0 conn(0x7f6910100920 legacy=0x7f691019cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:13:19.122 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:19.121+0000 7f6915091700 1 -- 172.21.15.183:0/827647761 shutdown_connections 2024-03-26T04:13:19.122 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:19.121+0000 7f6915091700 1 -- 172.21.15.183:0/827647761 >> 172.21.15.183:0/827647761 conn(0x7f69100fc360 msgr2=0x7f69101016a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:13:19.122 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:19.122+0000 7f6915091700 1 -- 172.21.15.183:0/827647761 shutdown_connections 2024-03-26T04:13:19.123 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:19.122+0000 7f6915091700 1 -- 172.21.15.183:0/827647761 wait complete. 2024-03-26T04:13:19.123 INFO:teuthology.orchestra.run.smithi183.stderr:dumped monmap epoch 1 2024-03-26T04:13:19.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:19 smithi183 systemd[1]: Starting Ceph mon.b for d7d59826-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:13:19.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:19 smithi016 ceph-mon[29101]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:19.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:19 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.183:0/827647761' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T04:13:20.082 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:19 smithi183 podman[33726]: 2024-03-26 04:13:19.73456366 +0000 UTC m=+0.017716320 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T04:13:20.082 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:19 smithi183 podman[33726]: 2024-03-26 04:13:19.837207634 +0000 UTC m=+0.120360336 container create 2b01545a9662c3b2398e36cdb75d36131476fb1cbb0f2ce776865f5407309650 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mon-b, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T04:13:20.082 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 podman[33726]: 2024-03-26 04:13:20.047717619 +0000 UTC m=+0.330870279 container init 2b01545a9662c3b2398e36cdb75d36131476fb1cbb0f2ce776865f5407309650 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mon-b, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T04:13:20.082 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 podman[33726]: 2024-03-26 04:13:20.05080981 +0000 UTC m=+0.333962471 container start 2b01545a9662c3b2398e36cdb75d36131476fb1cbb0f2ce776865f5407309650 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mon-b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T04:13:20.082 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T04:13:20.083 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev), process ceph-mon, pid 2 2024-03-26T04:13:20.083 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: pidfile_write: ignore empty --pid-file 2024-03-26T04:13:20.083 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: load: jerasure load: lrc 2024-03-26T04:13:20.083 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: RocksDB version: 7.9.2 2024-03-26T04:13:20.083 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Git sha 0 2024-03-26T04:13:20.083 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Compile date 2024-03-25 16:14:20 2024-03-26T04:13:20.083 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: DB SUMMARY 2024-03-26T04:13:20.083 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: DB Session ID: Y7POE5TJK2IMG66573QH 2024-03-26T04:13:20.083 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: CURRENT file: CURRENT 2024-03-26T04:13:20.083 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T04:13:20.083 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-26T04:13:20.083 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-26T04:13:20.083 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 476 ; 2024-03-26T04:13:20.085 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.error_if_exists: 0 2024-03-26T04:13:20.085 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.create_if_missing: 0 2024-03-26T04:13:20.085 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.paranoid_checks: 1 2024-03-26T04:13:20.085 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-26T04:13:20.085 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T04:13:20.085 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-26T04:13:20.085 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.env: 0x56503a2f6ae0 2024-03-26T04:13:20.085 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.fs: PosixFileSystem 2024-03-26T04:13:20.085 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.info_log: 0x56503b33d960 2024-03-26T04:13:20.085 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T04:13:20.085 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.statistics: (nil) 2024-03-26T04:13:20.085 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.use_fsync: 0 2024-03-26T04:13:20.085 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.max_log_file_size: 0 2024-03-26T04:13:20.086 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T04:13:20.086 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T04:13:20.086 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T04:13:20.086 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T04:13:20.086 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.allow_fallocate: 1 2024-03-26T04:13:20.086 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T04:13:20.086 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T04:13:20.086 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.use_direct_reads: 0 2024-03-26T04:13:20.086 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T04:13:20.086 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T04:13:20.086 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.db_log_dir: 2024-03-26T04:13:20.086 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.wal_dir: 2024-03-26T04:13:20.086 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T04:13:20.087 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T04:13:20.087 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T04:13:20.087 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T04:13:20.087 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T04:13:20.087 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T04:13:20.087 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T04:13:20.087 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T04:13:20.087 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.write_buffer_manager: 0x56503b3101e0 2024-03-26T04:13:20.087 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T04:13:20.087 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T04:13:20.087 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T04:13:20.087 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T04:13:20.087 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T04:13:20.088 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T04:13:20.088 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T04:13:20.088 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T04:13:20.088 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.unordered_write: 0 2024-03-26T04:13:20.088 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T04:13:20.088 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T04:13:20.088 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T04:13:20.088 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T04:13:20.088 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.row_cache: None 2024-03-26T04:13:20.088 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.wal_filter: None 2024-03-26T04:13:20.088 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T04:13:20.088 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T04:13:20.088 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.two_write_queues: 0 2024-03-26T04:13:20.088 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T04:13:20.088 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.wal_compression: 0 2024-03-26T04:13:20.089 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.atomic_flush: 0 2024-03-26T04:13:20.089 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T04:13:20.089 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T04:13:20.089 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T04:13:20.089 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.log_readahead_size: 0 2024-03-26T04:13:20.089 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T04:13:20.089 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T04:13:20.089 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T04:13:20.089 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T04:13:20.089 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T04:13:20.089 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T04:13:20.089 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T04:13:20.089 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.max_background_jobs: 2 2024-03-26T04:13:20.089 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.max_background_compactions: -1 2024-03-26T04:13:20.089 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.max_subcompactions: 1 2024-03-26T04:13:20.090 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T04:13:20.090 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T04:13:20.090 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T04:13:20.090 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T04:13:20.090 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T04:13:20.090 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T04:13:20.090 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T04:13:20.090 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T04:13:20.090 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.max_open_files: -1 2024-03-26T04:13:20.090 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T04:13:20.090 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T04:13:20.090 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T04:13:20.090 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T04:13:20.091 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.max_background_flushes: -1 2024-03-26T04:13:20.091 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Compression algorithms supported: 2024-03-26T04:13:20.091 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T04:13:20.091 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: kZSTD supported: 0 2024-03-26T04:13:20.091 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: kXpressCompression supported: 0 2024-03-26T04:13:20.091 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T04:13:20.091 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: kZlibCompression supported: 1 2024-03-26T04:13:20.091 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: kSnappyCompression supported: 1 2024-03-26T04:13:20.091 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: kLZ4Compression supported: 1 2024-03-26T04:13:20.091 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: kBZip2Compression supported: 0 2024-03-26T04:13:20.091 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T04:13:20.092 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T04:13:20.092 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-26T04:13:20.092 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T04:13:20.092 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T04:13:20.092 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.merge_operator: 2024-03-26T04:13:20.092 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.compaction_filter: None 2024-03-26T04:13:20.092 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T04:13:20.092 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T04:13:20.092 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T04:13:20.092 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T04:13:20.092 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56503b33da60) 2024-03-26T04:13:20.092 INFO:journalctl@ceph.mon.b.smithi183.stdout: cache_index_and_filter_blocks: 1 2024-03-26T04:13:20.093 INFO:journalctl@ceph.mon.b.smithi183.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T04:13:20.093 INFO:journalctl@ceph.mon.b.smithi183.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T04:13:20.093 INFO:journalctl@ceph.mon.b.smithi183.stdout: pin_top_level_index_and_filter: 1 2024-03-26T04:13:20.093 INFO:journalctl@ceph.mon.b.smithi183.stdout: index_type: 0 2024-03-26T04:13:20.093 INFO:journalctl@ceph.mon.b.smithi183.stdout: data_block_index_type: 0 2024-03-26T04:13:20.093 INFO:journalctl@ceph.mon.b.smithi183.stdout: index_shortening: 1 2024-03-26T04:13:20.093 INFO:journalctl@ceph.mon.b.smithi183.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T04:13:20.093 INFO:journalctl@ceph.mon.b.smithi183.stdout: checksum: 4 2024-03-26T04:13:20.093 INFO:journalctl@ceph.mon.b.smithi183.stdout: no_block_cache: 0 2024-03-26T04:13:20.093 INFO:journalctl@ceph.mon.b.smithi183.stdout: block_cache: 0x56503b341090 2024-03-26T04:13:20.093 INFO:journalctl@ceph.mon.b.smithi183.stdout: block_cache_name: BinnedLRUCache 2024-03-26T04:13:20.093 INFO:journalctl@ceph.mon.b.smithi183.stdout: block_cache_options: 2024-03-26T04:13:20.093 INFO:journalctl@ceph.mon.b.smithi183.stdout: capacity : 536870912 2024-03-26T04:13:20.093 INFO:journalctl@ceph.mon.b.smithi183.stdout: num_shard_bits : 4 2024-03-26T04:13:20.094 INFO:journalctl@ceph.mon.b.smithi183.stdout: strict_capacity_limit : 0 2024-03-26T04:13:20.094 INFO:journalctl@ceph.mon.b.smithi183.stdout: high_pri_pool_ratio: 0.000 2024-03-26T04:13:20.094 INFO:journalctl@ceph.mon.b.smithi183.stdout: block_cache_compressed: (nil) 2024-03-26T04:13:20.094 INFO:journalctl@ceph.mon.b.smithi183.stdout: persistent_cache: (nil) 2024-03-26T04:13:20.094 INFO:journalctl@ceph.mon.b.smithi183.stdout: block_size: 4096 2024-03-26T04:13:20.094 INFO:journalctl@ceph.mon.b.smithi183.stdout: block_size_deviation: 10 2024-03-26T04:13:20.094 INFO:journalctl@ceph.mon.b.smithi183.stdout: block_restart_interval: 16 2024-03-26T04:13:20.094 INFO:journalctl@ceph.mon.b.smithi183.stdout: index_block_restart_interval: 1 2024-03-26T04:13:20.094 INFO:journalctl@ceph.mon.b.smithi183.stdout: metadata_block_size: 4096 2024-03-26T04:13:20.094 INFO:journalctl@ceph.mon.b.smithi183.stdout: partition_filters: 0 2024-03-26T04:13:20.094 INFO:journalctl@ceph.mon.b.smithi183.stdout: use_delta_encoding: 1 2024-03-26T04:13:20.094 INFO:journalctl@ceph.mon.b.smithi183.stdout: filter_policy: bloomfilter 2024-03-26T04:13:20.094 INFO:journalctl@ceph.mon.b.smithi183.stdout: whole_key_filtering: 1 2024-03-26T04:13:20.095 INFO:journalctl@ceph.mon.b.smithi183.stdout: verify_compression: 0 2024-03-26T04:13:20.095 INFO:journalctl@ceph.mon.b.smithi183.stdout: read_amp_bytes_per_bit: 0 2024-03-26T04:13:20.095 INFO:journalctl@ceph.mon.b.smithi183.stdout: format_version: 5 2024-03-26T04:13:20.095 INFO:journalctl@ceph.mon.b.smithi183.stdout: enable_index_compression: 1 2024-03-26T04:13:20.095 INFO:journalctl@ceph.mon.b.smithi183.stdout: block_align: 0 2024-03-26T04:13:20.095 INFO:journalctl@ceph.mon.b.smithi183.stdout: max_auto_readahead_size: 262144 2024-03-26T04:13:20.095 INFO:journalctl@ceph.mon.b.smithi183.stdout: prepopulate_block_cache: 0 2024-03-26T04:13:20.095 INFO:journalctl@ceph.mon.b.smithi183.stdout: initial_auto_readahead_size: 8192 2024-03-26T04:13:20.095 INFO:journalctl@ceph.mon.b.smithi183.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T04:13:20.095 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T04:13:20.095 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T04:13:20.095 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.compression: NoCompression 2024-03-26T04:13:20.095 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T04:13:20.095 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T04:13:20.096 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T04:13:20.096 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.num_levels: 7 2024-03-26T04:13:20.096 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T04:13:20.096 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T04:13:20.096 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T04:13:20.096 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T04:13:20.096 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T04:13:20.096 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T04:13:20.096 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T04:13:20.096 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T04:13:20.096 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T04:13:20.096 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T04:13:20.096 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T04:13:20.096 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T04:13:20.097 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T04:13:20.097 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T04:13:20.097 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T04:13:20.097 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T04:13:20.097 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T04:13:20.097 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T04:13:20.097 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T04:13:20.097 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T04:13:20.097 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T04:13:20.097 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T04:13:20.097 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T04:13:20.097 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T04:13:20.097 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T04:13:20.097 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T04:13:20.098 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T04:13:20.098 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T04:13:20.098 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T04:13:20.098 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T04:13:20.098 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T04:13:20.098 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T04:13:20.098 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T04:13:20.098 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T04:13:20.098 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T04:13:20.098 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T04:13:20.098 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T04:13:20.098 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T04:13:20.098 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T04:13:20.099 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T04:13:20.099 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T04:13:20.099 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T04:13:20.099 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T04:13:20.099 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T04:13:20.099 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T04:13:20.099 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T04:13:20.099 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T04:13:20.099 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T04:13:20.099 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T04:13:20.099 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T04:13:20.099 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T04:13:20.099 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T04:13:20.099 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T04:13:20.099 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.table_properties_collectors: 2024-03-26T04:13:20.100 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.inplace_update_support: 0 2024-03-26T04:13:20.100 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T04:13:20.100 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T04:13:20.100 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T04:13:20.100 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T04:13:20.101 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.bloom_locality: 0 2024-03-26T04:13:20.101 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.max_successive_merges: 0 2024-03-26T04:13:20.101 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T04:13:20.101 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T04:13:20.101 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T04:13:20.101 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T04:13:20.101 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.ttl: 2592000 2024-03-26T04:13:20.101 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T04:13:20.101 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T04:13:20.101 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T04:13:20.101 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.enable_blob_files: false 2024-03-26T04:13:20.101 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.min_blob_size: 0 2024-03-26T04:13:20.101 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T04:13:20.101 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T04:13:20.102 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T04:13:20.102 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T04:13:20.102 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T04:13:20.102 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T04:13:20.102 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T04:13:20.102 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T04:13:20.102 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-26T04:13:20.102 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-26T04:13:20.102 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 5295d492-a55c-4357-ab7e-93632275c7dc 2024-03-26T04:13:20.102 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711426400080280, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-26T04:13:20.102 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-26T04:13:20.102 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711426400080689, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1587, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 488, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 366, "raw_average_value_size": 73, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711426400, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "5295d492-a55c-4357-ab7e-93632275c7dc", "db_session_id": "Y7POE5TJK2IMG66573QH", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-26T04:13:20.103 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711426400080767, "job": 1, "event": "recovery_finished"} 2024-03-26T04:13:20.103 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-26T04:13:20.103 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T04:13:20.103 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56503b432000 2024-03-26T04:13:20.103 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: DB pointer 0x56503b41c000 2024-03-26T04:13:20.103 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T04:13:20.103 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T04:13:20.103 INFO:journalctl@ceph.mon.b.smithi183.stdout: ** DB Stats ** 2024-03-26T04:13:20.103 INFO:journalctl@ceph.mon.b.smithi183.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T04:13:20.103 INFO:journalctl@ceph.mon.b.smithi183.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T04:13:20.103 INFO:journalctl@ceph.mon.b.smithi183.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T04:13:20.103 INFO:journalctl@ceph.mon.b.smithi183.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T04:13:20.103 INFO:journalctl@ceph.mon.b.smithi183.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T04:13:20.103 INFO:journalctl@ceph.mon.b.smithi183.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T04:13:20.104 INFO:journalctl@ceph.mon.b.smithi183.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T04:13:20.104 INFO:journalctl@ceph.mon.b.smithi183.stdout: 2024-03-26T04:13:20.104 INFO:journalctl@ceph.mon.b.smithi183.stdout: ** Compaction Stats [default] ** 2024-03-26T04:13:20.104 INFO:journalctl@ceph.mon.b.smithi183.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T04:13:20.104 INFO:journalctl@ceph.mon.b.smithi183.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T04:13:20.104 INFO:journalctl@ceph.mon.b.smithi183.stdout: L0 1/0 1.55 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T04:13:20.104 INFO:journalctl@ceph.mon.b.smithi183.stdout: Sum 1/0 1.55 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T04:13:20.104 INFO:journalctl@ceph.mon.b.smithi183.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T04:13:20.104 INFO:journalctl@ceph.mon.b.smithi183.stdout: 2024-03-26T04:13:20.104 INFO:journalctl@ceph.mon.b.smithi183.stdout: ** Compaction Stats [default] ** 2024-03-26T04:13:20.104 INFO:journalctl@ceph.mon.b.smithi183.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T04:13:20.104 INFO:journalctl@ceph.mon.b.smithi183.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T04:13:20.104 INFO:journalctl@ceph.mon.b.smithi183.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T04:13:20.104 INFO:journalctl@ceph.mon.b.smithi183.stdout: 2024-03-26T04:13:20.105 INFO:journalctl@ceph.mon.b.smithi183.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T04:13:20.105 INFO:journalctl@ceph.mon.b.smithi183.stdout: 2024-03-26T04:13:20.105 INFO:journalctl@ceph.mon.b.smithi183.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T04:13:20.105 INFO:journalctl@ceph.mon.b.smithi183.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-26T04:13:20.105 INFO:journalctl@ceph.mon.b.smithi183.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T04:13:20.105 INFO:journalctl@ceph.mon.b.smithi183.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T04:13:20.105 INFO:journalctl@ceph.mon.b.smithi183.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T04:13:20.105 INFO:journalctl@ceph.mon.b.smithi183.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T04:13:20.105 INFO:journalctl@ceph.mon.b.smithi183.stdout: Cumulative compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T04:13:20.105 INFO:journalctl@ceph.mon.b.smithi183.stdout: Interval compaction: 0.00 GB write, 0.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T04:13:20.105 INFO:journalctl@ceph.mon.b.smithi183.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T04:13:20.105 INFO:journalctl@ceph.mon.b.smithi183.stdout: Block cache BinnedLRUCache@0x56503b341090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-26T04:13:20.105 INFO:journalctl@ceph.mon.b.smithi183.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-26T04:13:20.106 INFO:journalctl@ceph.mon.b.smithi183.stdout: 2024-03-26T04:13:20.106 INFO:journalctl@ceph.mon.b.smithi183.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T04:13:20.106 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-26T04:13:20.106 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: using public_addrv v1:172.21.15.183:6789/0 2024-03-26T04:13:20.106 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: starting mon.b rank -1 at public addrs v1:172.21.15.183:6789/0 at bind addrs v1:172.21.15.183:6789/0 mon_data /var/lib/ceph/mon/ceph-b fsid d7d59826-eb26-11ee-95d0-87774f69a715 2024-03-26T04:13:20.106 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: mon.b@-1(???) e0 preinit fsid d7d59826-eb26-11ee-95d0-87774f69a715 2024-03-26T04:13:20.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: mon.b@-1(synchronizing).mds e1 new map 2024-03-26T04:13:20.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-26T04:13:20.542 INFO:journalctl@ceph.mon.b.smithi183.stdout: e1 2024-03-26T04:13:20.542 INFO:journalctl@ceph.mon.b.smithi183.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T04:13:20.542 INFO:journalctl@ceph.mon.b.smithi183.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-26T04:13:20.542 INFO:journalctl@ceph.mon.b.smithi183.stdout: legacy client fscid: -1 2024-03-26T04:13:20.542 INFO:journalctl@ceph.mon.b.smithi183.stdout: 2024-03-26T04:13:20.543 INFO:journalctl@ceph.mon.b.smithi183.stdout: No filesystems configured 2024-03-26T04:13:20.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-26T04:13:20.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-26T04:13:20.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-26T04:13:20.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-26T04:13:20.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-26T04:13:20.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-26T04:13:20.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-26T04:13:20.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T04:13:20.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T04:13:20.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T04:13:20.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:20.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:20.544 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: from='client.14180 v1:172.21.15.16:0/3695379932' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:13:20.544 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:20.544 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1934386500' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-26T04:13:20.544 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:20.544 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:20.544 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:20.544 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:20.544 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi183", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:13:20.544 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:20.544 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:13:20.544 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: Updating smithi183:/etc/ceph/ceph.conf 2024-03-26T04:13:20.544 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: Updating smithi183:/var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:13:20.544 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:20.544 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: Updating smithi183:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:13:20.545 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1934386500' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-26T04:13:20.545 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:13:20.545 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: Updating smithi183:/var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:13:20.545 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:20.545 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:20.545 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:20.545 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:20.545 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: from='client.14184 v1:172.21.15.183:0/3681984561' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi016:[v1:172.21.15.16:6789]=a;smithi016:[v1:172.21.15.16:6790]=c;smithi183:[v1:172.21.15.183:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:13:20.545 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: Saving service mon spec with placement smithi016:[v1:172.21.15.16:6789]=a;smithi016:[v1:172.21.15.16:6790]=c;smithi183:[v1:172.21.15.183:6789]=b;count:3 2024-03-26T04:13:20.545 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:20.545 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:13:20.545 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:20.545 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:13:20.546 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:20.546 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:13:20.546 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:20.546 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: Deploying daemon mon.b on smithi183 2024-03-26T04:13:20.546 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:20.546 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:20.546 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.183:0/827647761' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T04:13:20.546 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 ceph-mon[33745]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-26T04:13:20.546 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 bash[33726]: 2b01545a9662c3b2398e36cdb75d36131476fb1cbb0f2ce776865f5407309650 2024-03-26T04:13:20.546 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:20 smithi183 systemd[1]: Started Ceph mon.b for d7d59826-eb26-11ee-95d0-87774f69a715. 2024-03-26T04:13:20.647 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T04:13:20.648 DEBUG:teuthology.orchestra.run.smithi183:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T04:13:20.889 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:13:25.151 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:25.148+0000 7ffa5fb19700 1 -- 172.21.15.183:0/1212394188 >> v1:172.21.15.16:6789/0 conn(0x7ffa581069d0 legacy=0x7ffa58106db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:13:25.151 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:25.148+0000 7ffa5fb19700 1 -- 172.21.15.183:0/1212394188 shutdown_connections 2024-03-26T04:13:25.151 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:25.149+0000 7ffa5fb19700 1 -- 172.21.15.183:0/1212394188 >> 172.21.15.183:0/1212394188 conn(0x7ffa580fc4e0 msgr2=0x7ffa580fe900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:13:25.151 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:25.149+0000 7ffa5fb19700 1 -- 172.21.15.183:0/1212394188 shutdown_connections 2024-03-26T04:13:25.151 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:25.149+0000 7ffa5fb19700 1 -- 172.21.15.183:0/1212394188 wait complete. 2024-03-26T04:13:25.151 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:25.149+0000 7ffa5fb19700 1 Processor -- start 2024-03-26T04:13:25.151 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:25.150+0000 7ffa5fb19700 1 -- start start 2024-03-26T04:13:25.151 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:25.150+0000 7ffa5fb19700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ffa580fd550 con 0x7ffa581069d0 2024-03-26T04:13:25.151 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:25.150+0000 7ffa5fb19700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ffa580fd6e0 con 0x7ffa580fd0a0 2024-03-26T04:13:25.151 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:25.150+0000 7ffa5d0b4700 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7ffa580fd0a0 0x7ffa581a39f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:58948/0 (socket says 172.21.15.183:58948) 2024-03-26T04:13:25.151 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:25.150+0000 7ffa5d0b4700 1 -- 172.21.15.183:0/1141374543 learned_addr learned my addr 172.21.15.183:0/1141374543 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-26T04:13:25.152 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:25.150+0000 7ffa4effd700 1 -- 172.21.15.183:0/1141374543 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2210015406 0 0) 0x7ffa580fd550 con 0x7ffa581069d0 2024-03-26T04:13:25.152 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:25.151+0000 7ffa4effd700 1 -- 172.21.15.183:0/1141374543 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa440035b0 con 0x7ffa581069d0 2024-03-26T04:13:25.152 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:25.151+0000 7ffa4effd700 1 -- 172.21.15.183:0/1141374543 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 283790375 0 0) 0x7ffa440035b0 con 0x7ffa581069d0 2024-03-26T04:13:25.152 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:25.151+0000 7ffa4effd700 1 -- 172.21.15.183:0/1141374543 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa580fd550 con 0x7ffa581069d0 2024-03-26T04:13:25.153 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:25.151+0000 7ffa4effd700 1 -- 172.21.15.183:0/1141374543 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 239+0+0 (unknown 40298983 0 0) 0x7ffa50004830 con 0x7ffa581069d0 2024-03-26T04:13:25.153 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:25.151+0000 7ffa4effd700 1 -- 172.21.15.183:0/1141374543 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1119750025 0 0) 0x7ffa580fd550 con 0x7ffa581069d0 2024-03-26T04:13:25.153 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:25.151+0000 7ffa4effd700 1 -- 172.21.15.183:0/1141374543 >> v1:172.21.15.183:6789/0 conn(0x7ffa580fd0a0 legacy=0x7ffa581a39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:13:25.153 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:25.151+0000 7ffa4effd700 1 -- 172.21.15.183:0/1141374543 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa581a7130 con 0x7ffa581069d0 2024-03-26T04:13:25.153 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:25.151+0000 7ffa5fb19700 1 -- 172.21.15.183:0/1141374543 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa581a5110 con 0x7ffa581069d0 2024-03-26T04:13:25.153 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:25.152+0000 7ffa5fb19700 1 -- 172.21.15.183:0/1141374543 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa581a55d0 con 0x7ffa581069d0 2024-03-26T04:13:25.153 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:25.152+0000 7ffa4effd700 1 -- 172.21.15.183:0/1141374543 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7ffa50001e70 con 0x7ffa581069d0 2024-03-26T04:13:25.153 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:25.152+0000 7ffa4effd700 1 -- 172.21.15.183:0/1141374543 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 239+0+0 (unknown 40298983 0 0) 0x7ffa500051e0 con 0x7ffa581069d0 2024-03-26T04:13:25.154 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:25.152+0000 7ffa4effd700 1 -- 172.21.15.183:0/1141374543 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 13) v1 ==== 45949+0+0 (unknown 1805086598 0 0) 0x7ffa500054b0 con 0x7ffa581069d0 2024-03-26T04:13:25.154 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:25.153+0000 7ffa4effd700 1 -- 172.21.15.183:0/1141374543 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 1069+0+0 (unknown 21079357 0 0) 0x7ffa500474a0 con 0x7ffa581069d0 2024-03-26T04:13:25.154 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:25.153+0000 7ffa5fb19700 1 -- 172.21.15.183:0/1141374543 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa58102110 con 0x7ffa581069d0 2024-03-26T04:13:25.160 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:25.160+0000 7ffa4effd700 1 -- 172.21.15.183:0/1141374543 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7ffa50005fe0 con 0x7ffa581069d0 2024-03-26T04:13:25.437 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:25.436+0000 7ffa5fb19700 1 -- 172.21.15.183:0/1141374543 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffa580fdec0 con 0x7ffa581069d0 2024-03-26T04:13:25.438 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:25 smithi183 ceph-mon[33745]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:25.438 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:25 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:13:25.438 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:25 smithi183 ceph-mon[33745]: mon.a calling monitor election 2024-03-26T04:13:25.438 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:25 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:13:25.438 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:25 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:13:25.438 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:25 smithi183 ceph-mon[33745]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:25.438 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:25 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:13:25.438 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:25 smithi183 ceph-mon[33745]: mon.b calling monitor election 2024-03-26T04:13:25.438 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:25 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:13:25.438 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:25 smithi183 ceph-mon[33745]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:25.438 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:25 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:13:25.438 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:25 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:13:25.438 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:25 smithi183 ceph-mon[33745]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-26T04:13:25.439 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:25 smithi183 ceph-mon[33745]: monmap e2: 2 mons at {a=v1:172.21.15.16:6789/0,b=v1:172.21.15.183:6789/0} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:13:25.439 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:25 smithi183 ceph-mon[33745]: fsmap 2024-03-26T04:13:25.439 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:25 smithi183 ceph-mon[33745]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:13:25.439 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:25 smithi183 ceph-mon[33745]: mgrmap e13: y(active, since 41s) 2024-03-26T04:13:25.439 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:25 smithi183 ceph-mon[33745]: overall HEALTH_OK 2024-03-26T04:13:25.439 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:25 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:25.439 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:25 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:25.439 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:25 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:25.439 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:25 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:13:25.439 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:25 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:25.439 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:25.437+0000 7ffa4effd700 1 -- 172.21.15.183:0/1141374543 <== mon.0 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+912 (unknown 3557084514 0 2870521719) 0x7ffa50015b00 con 0x7ffa581069d0 2024-03-26T04:13:25.440 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:13:25.440 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":2,"fsid":"d7d59826-eb26-11ee-95d0-87774f69a715","modified":"2024-03-26T04:13:20.101333Z","created":"2024-03-26T04:11:44.857505Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6789","nonce":0}]},"addr":"172.21.15.183:6789/0","public_addr":"172.21.15.183:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-26T04:13:25.440 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:25.439+0000 7ffa5fb19700 1 -- 172.21.15.183:0/1141374543 >> v1:172.21.15.16:6800/4042489921 conn(0x7ffa4403a2b0 legacy=0x7ffa4403c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:13:25.441 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:25.439+0000 7ffa5fb19700 1 -- 172.21.15.183:0/1141374543 >> v1:172.21.15.16:6789/0 conn(0x7ffa581069d0 legacy=0x7ffa580fc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:13:25.441 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:25.439+0000 7ffa5fb19700 1 -- 172.21.15.183:0/1141374543 shutdown_connections 2024-03-26T04:13:25.441 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:25.439+0000 7ffa5fb19700 1 -- 172.21.15.183:0/1141374543 >> 172.21.15.183:0/1141374543 conn(0x7ffa580fc4e0 msgr2=0x7ffa5806a350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:13:25.441 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:25.439+0000 7ffa5fb19700 1 -- 172.21.15.183:0/1141374543 shutdown_connections 2024-03-26T04:13:25.441 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:25.439+0000 7ffa5fb19700 1 -- 172.21.15.183:0/1141374543 wait complete. 2024-03-26T04:13:25.441 INFO:teuthology.orchestra.run.smithi183.stderr:dumped monmap epoch 2 2024-03-26T04:13:25.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:25 smithi016 ceph-mon[29101]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:25.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:25 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:13:25.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:25 smithi016 ceph-mon[29101]: mon.a calling monitor election 2024-03-26T04:13:25.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:25 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:13:25.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:25 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:13:25.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:25 smithi016 ceph-mon[29101]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:25.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:25 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:13:25.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:25 smithi016 ceph-mon[29101]: mon.b calling monitor election 2024-03-26T04:13:25.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:25 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:13:25.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:25 smithi016 ceph-mon[29101]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:25.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:25 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:13:25.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:25 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:13:25.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:25 smithi016 ceph-mon[29101]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-26T04:13:25.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:25 smithi016 ceph-mon[29101]: monmap e2: 2 mons at {a=v1:172.21.15.16:6789/0,b=v1:172.21.15.183:6789/0} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:13:25.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:25 smithi016 ceph-mon[29101]: fsmap 2024-03-26T04:13:25.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:25 smithi016 ceph-mon[29101]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:13:25.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:25 smithi016 ceph-mon[29101]: mgrmap e13: y(active, since 41s) 2024-03-26T04:13:25.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:25 smithi016 ceph-mon[29101]: overall HEALTH_OK 2024-03-26T04:13:25.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:25 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:25.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:25 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:25.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:25 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:25.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:25 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:13:25.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:25 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:26.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:26 smithi183 ceph-mon[33745]: Deploying daemon mon.c on smithi016 2024-03-26T04:13:26.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:26 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.183:0/1141374543' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T04:13:26.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:26 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:13:26.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:26 smithi016 ceph-mon[29101]: Deploying daemon mon.c on smithi016 2024-03-26T04:13:26.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:26 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.183:0/1141374543' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T04:13:26.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:26 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:13:27.030 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-26T04:13:27.030 DEBUG:teuthology.orchestra.run.smithi183:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph mon dump -f json 2024-03-26T04:13:27.277 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:13:27.398 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:27 smithi183 ceph-mon[33745]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:27.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:27 smithi016 ceph-mon[29101]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:27.412 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:13:27 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:13:27.101+0000 7fdf4d229700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-26T04:13:28.303 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 systemd[1]: Starting Ceph mon.c for d7d59826-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:13:28.589 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 podman[37196]: 2024-03-26 04:13:28.303632228 +0000 UTC m=+0.092284596 container create 2d20f53bd9c70a6c1116fe28f7ff96f3ff67dc8a3984598780567611f352f49c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-26T04:13:28.589 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 podman[37196]: 2024-03-26 04:13:28.22858876 +0000 UTC m=+0.017241127 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T04:13:28.589 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 podman[37196]: 2024-03-26 04:13:28.589293945 +0000 UTC m=+0.377946313 container init 2d20f53bd9c70a6c1116fe28f7ff96f3ff67dc8a3984598780567611f352f49c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mon-c, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-26T04:13:28.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 podman[37196]: 2024-03-26 04:13:28.592298794 +0000 UTC m=+0.380951163 container start 2d20f53bd9c70a6c1116fe28f7ff96f3ff67dc8a3984598780567611f352f49c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mon-c, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T04:13:28.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: set uid:gid to 167:167 (ceph:ceph) 2024-03-26T04:13:28.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: ceph version 19.0.0-1775-g5e70124b (5e70124b46c2a671efcaae4690cd9ea72670710a) squid (dev), process ceph-mon, pid 2 2024-03-26T04:13:28.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: pidfile_write: ignore empty --pid-file 2024-03-26T04:13:28.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: load: jerasure load: lrc 2024-03-26T04:13:28.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: RocksDB version: 7.9.2 2024-03-26T04:13:28.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Git sha 0 2024-03-26T04:13:28.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Compile date 2024-03-25 16:14:20 2024-03-26T04:13:28.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: DB SUMMARY 2024-03-26T04:13:28.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: DB Session ID: 2PSUGTFMF5262059B06B 2024-03-26T04:13:28.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: CURRENT file: CURRENT 2024-03-26T04:13:28.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: IDENTITY file: IDENTITY 2024-03-26T04:13:28.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-26T04:13:28.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-26T04:13:28.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 566 ; 2024-03-26T04:13:28.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.error_if_exists: 0 2024-03-26T04:13:28.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.create_if_missing: 0 2024-03-26T04:13:28.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.paranoid_checks: 1 2024-03-26T04:13:28.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-26T04:13:28.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-26T04:13:28.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-26T04:13:28.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.env: 0x5570b81a5ae0 2024-03-26T04:13:28.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.fs: PosixFileSystem 2024-03-26T04:13:28.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.info_log: 0x5570b9fb5920 2024-03-26T04:13:28.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.max_file_opening_threads: 16 2024-03-26T04:13:28.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.statistics: (nil) 2024-03-26T04:13:28.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.use_fsync: 0 2024-03-26T04:13:28.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.max_log_file_size: 0 2024-03-26T04:13:28.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-26T04:13:28.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-26T04:13:28.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.keep_log_file_num: 1000 2024-03-26T04:13:28.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.recycle_log_file_num: 0 2024-03-26T04:13:28.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.allow_fallocate: 1 2024-03-26T04:13:28.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.allow_mmap_reads: 0 2024-03-26T04:13:28.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.allow_mmap_writes: 0 2024-03-26T04:13:28.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.use_direct_reads: 0 2024-03-26T04:13:28.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-26T04:13:28.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.create_missing_column_families: 0 2024-03-26T04:13:28.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.db_log_dir: 2024-03-26T04:13:28.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.wal_dir: 2024-03-26T04:13:28.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-26T04:13:28.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-26T04:13:28.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-26T04:13:28.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-26T04:13:28.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-26T04:13:28.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-26T04:13:28.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.advise_random_on_open: 1 2024-03-26T04:13:28.917 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.db_write_buffer_size: 0 2024-03-26T04:13:28.917 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.write_buffer_manager: 0x5570b9f881e0 2024-03-26T04:13:28.917 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-26T04:13:28.917 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-26T04:13:28.917 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-26T04:13:28.917 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.rate_limiter: (nil) 2024-03-26T04:13:28.917 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-26T04:13:28.917 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.wal_recovery_mode: 2 2024-03-26T04:13:28.917 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.enable_thread_tracking: 0 2024-03-26T04:13:28.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.enable_pipelined_write: 0 2024-03-26T04:13:28.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.unordered_write: 0 2024-03-26T04:13:28.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-26T04:13:28.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-26T04:13:28.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-26T04:13:28.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-26T04:13:28.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.row_cache: None 2024-03-26T04:13:28.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.wal_filter: None 2024-03-26T04:13:28.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-26T04:13:28.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.allow_ingest_behind: 0 2024-03-26T04:13:28.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.two_write_queues: 0 2024-03-26T04:13:28.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.manual_wal_flush: 0 2024-03-26T04:13:28.919 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.wal_compression: 0 2024-03-26T04:13:28.919 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.atomic_flush: 0 2024-03-26T04:13:28.919 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-26T04:13:28.919 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-26T04:13:28.919 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-26T04:13:28.922 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.log_readahead_size: 0 2024-03-26T04:13:28.922 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-26T04:13:28.922 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.best_efforts_recovery: 0 2024-03-26T04:13:28.922 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-26T04:13:28.922 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-26T04:13:28.923 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.allow_data_in_errors: 0 2024-03-26T04:13:28.923 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.db_host_id: __hostname__ 2024-03-26T04:13:28.923 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-26T04:13:28.923 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.max_background_jobs: 2 2024-03-26T04:13:28.923 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.max_background_compactions: -1 2024-03-26T04:13:28.924 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.max_subcompactions: 1 2024-03-26T04:13:28.924 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-26T04:13:28.924 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-26T04:13:28.924 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-26T04:13:28.924 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.max_total_wal_size: 0 2024-03-26T04:13:28.924 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-26T04:13:28.925 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-26T04:13:28.925 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-26T04:13:28.925 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-26T04:13:28.925 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.max_open_files: -1 2024-03-26T04:13:28.925 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.bytes_per_sync: 0 2024-03-26T04:13:28.925 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-26T04:13:28.926 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-26T04:13:28.926 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.compaction_readahead_size: 0 2024-03-26T04:13:28.926 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.max_background_flushes: -1 2024-03-26T04:13:28.926 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Compression algorithms supported: 2024-03-26T04:13:28.926 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-26T04:13:28.926 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: kZSTD supported: 0 2024-03-26T04:13:28.927 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: kXpressCompression supported: 0 2024-03-26T04:13:28.927 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: kLZ4HCCompression supported: 1 2024-03-26T04:13:28.927 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: kZlibCompression supported: 1 2024-03-26T04:13:28.927 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: kSnappyCompression supported: 1 2024-03-26T04:13:28.927 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: kLZ4Compression supported: 1 2024-03-26T04:13:28.927 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: kBZip2Compression supported: 0 2024-03-26T04:13:28.928 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-26T04:13:28.928 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-26T04:13:28.928 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-26T04:13:28.928 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-26T04:13:28.928 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-26T04:13:28.928 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.merge_operator: 2024-03-26T04:13:28.929 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.compaction_filter: None 2024-03-26T04:13:28.929 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.compaction_filter_factory: None 2024-03-26T04:13:28.929 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.sst_partitioner_factory: None 2024-03-26T04:13:28.929 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-26T04:13:28.929 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-26T04:13:28.929 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5570b9fb5a20) 2024-03-26T04:13:28.929 INFO:journalctl@ceph.mon.c.smithi016.stdout: cache_index_and_filter_blocks: 1 2024-03-26T04:13:28.930 INFO:journalctl@ceph.mon.c.smithi016.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-26T04:13:28.930 INFO:journalctl@ceph.mon.c.smithi016.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-26T04:13:28.930 INFO:journalctl@ceph.mon.c.smithi016.stdout: pin_top_level_index_and_filter: 1 2024-03-26T04:13:28.930 INFO:journalctl@ceph.mon.c.smithi016.stdout: index_type: 0 2024-03-26T04:13:28.930 INFO:journalctl@ceph.mon.c.smithi016.stdout: data_block_index_type: 0 2024-03-26T04:13:28.930 INFO:journalctl@ceph.mon.c.smithi016.stdout: index_shortening: 1 2024-03-26T04:13:28.931 INFO:journalctl@ceph.mon.c.smithi016.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-26T04:13:28.931 INFO:journalctl@ceph.mon.c.smithi016.stdout: checksum: 4 2024-03-26T04:13:28.931 INFO:journalctl@ceph.mon.c.smithi016.stdout: no_block_cache: 0 2024-03-26T04:13:28.931 INFO:journalctl@ceph.mon.c.smithi016.stdout: block_cache: 0x5570b9fb9090 2024-03-26T04:13:28.931 INFO:journalctl@ceph.mon.c.smithi016.stdout: block_cache_name: BinnedLRUCache 2024-03-26T04:13:28.931 INFO:journalctl@ceph.mon.c.smithi016.stdout: block_cache_options: 2024-03-26T04:13:28.931 INFO:journalctl@ceph.mon.c.smithi016.stdout: capacity : 536870912 2024-03-26T04:13:28.932 INFO:journalctl@ceph.mon.c.smithi016.stdout: num_shard_bits : 4 2024-03-26T04:13:28.932 INFO:journalctl@ceph.mon.c.smithi016.stdout: strict_capacity_limit : 0 2024-03-26T04:13:28.932 INFO:journalctl@ceph.mon.c.smithi016.stdout: high_pri_pool_ratio: 0.000 2024-03-26T04:13:28.932 INFO:journalctl@ceph.mon.c.smithi016.stdout: block_cache_compressed: (nil) 2024-03-26T04:13:28.932 INFO:journalctl@ceph.mon.c.smithi016.stdout: persistent_cache: (nil) 2024-03-26T04:13:28.932 INFO:journalctl@ceph.mon.c.smithi016.stdout: block_size: 4096 2024-03-26T04:13:28.933 INFO:journalctl@ceph.mon.c.smithi016.stdout: block_size_deviation: 10 2024-03-26T04:13:28.933 INFO:journalctl@ceph.mon.c.smithi016.stdout: block_restart_interval: 16 2024-03-26T04:13:28.933 INFO:journalctl@ceph.mon.c.smithi016.stdout: index_block_restart_interval: 1 2024-03-26T04:13:28.933 INFO:journalctl@ceph.mon.c.smithi016.stdout: metadata_block_size: 4096 2024-03-26T04:13:28.933 INFO:journalctl@ceph.mon.c.smithi016.stdout: partition_filters: 0 2024-03-26T04:13:28.934 INFO:journalctl@ceph.mon.c.smithi016.stdout: use_delta_encoding: 1 2024-03-26T04:13:28.935 INFO:journalctl@ceph.mon.c.smithi016.stdout: filter_policy: bloomfilter 2024-03-26T04:13:28.935 INFO:journalctl@ceph.mon.c.smithi016.stdout: whole_key_filtering: 1 2024-03-26T04:13:28.935 INFO:journalctl@ceph.mon.c.smithi016.stdout: verify_compression: 0 2024-03-26T04:13:28.935 INFO:journalctl@ceph.mon.c.smithi016.stdout: read_amp_bytes_per_bit: 0 2024-03-26T04:13:28.935 INFO:journalctl@ceph.mon.c.smithi016.stdout: format_version: 5 2024-03-26T04:13:28.935 INFO:journalctl@ceph.mon.c.smithi016.stdout: enable_index_compression: 1 2024-03-26T04:13:28.935 INFO:journalctl@ceph.mon.c.smithi016.stdout: block_align: 0 2024-03-26T04:13:28.935 INFO:journalctl@ceph.mon.c.smithi016.stdout: max_auto_readahead_size: 262144 2024-03-26T04:13:28.935 INFO:journalctl@ceph.mon.c.smithi016.stdout: prepopulate_block_cache: 0 2024-03-26T04:13:28.935 INFO:journalctl@ceph.mon.c.smithi016.stdout: initial_auto_readahead_size: 8192 2024-03-26T04:13:28.935 INFO:journalctl@ceph.mon.c.smithi016.stdout: num_file_reads_for_auto_readahead: 2 2024-03-26T04:13:28.936 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.write_buffer_size: 33554432 2024-03-26T04:13:28.936 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.max_write_buffer_number: 2 2024-03-26T04:13:28.936 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.compression: NoCompression 2024-03-26T04:13:28.936 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.bottommost_compression: Disabled 2024-03-26T04:13:28.936 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.prefix_extractor: nullptr 2024-03-26T04:13:28.936 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-26T04:13:28.936 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.num_levels: 7 2024-03-26T04:13:28.936 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-26T04:13:28.936 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-26T04:13:28.937 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-26T04:13:28.937 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-26T04:13:28.937 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-26T04:13:28.937 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-26T04:13:28.937 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-26T04:13:28.937 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-26T04:13:28.937 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-26T04:13:28.937 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-26T04:13:28.937 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-26T04:13:28.937 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-26T04:13:28.937 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-26T04:13:28.937 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.compression_opts.level: 32767 2024-03-26T04:13:28.937 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.compression_opts.strategy: 0 2024-03-26T04:13:28.938 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-26T04:13:28.938 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-26T04:13:28.938 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-26T04:13:28.938 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-26T04:13:28.938 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.compression_opts.enabled: false 2024-03-26T04:13:28.938 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-26T04:13:28.938 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-26T04:13:28.938 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-26T04:13:28.938 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-26T04:13:28.938 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.target_file_size_base: 67108864 2024-03-26T04:13:28.938 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-26T04:13:28.938 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-26T04:13:28.939 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-26T04:13:28.939 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-26T04:13:28.939 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-26T04:13:28.939 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-26T04:13:28.939 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-26T04:13:28.939 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-26T04:13:28.939 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-26T04:13:28.939 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-26T04:13:28.939 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-26T04:13:28.939 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-26T04:13:28.939 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-26T04:13:28.939 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-26T04:13:28.940 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.arena_block_size: 1048576 2024-03-26T04:13:28.940 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-26T04:13:28.940 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-26T04:13:28.940 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.disable_auto_compactions: 0 2024-03-26T04:13:28.940 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-26T04:13:28.940 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-26T04:13:28.940 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-26T04:13:28.940 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-26T04:13:28.940 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-26T04:13:28.940 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-26T04:13:28.940 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-26T04:13:28.940 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-26T04:13:28.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-26T04:13:28.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-26T04:13:28.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.table_properties_collectors: 2024-03-26T04:13:28.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.inplace_update_support: 0 2024-03-26T04:13:28.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-26T04:13:28.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-26T04:13:28.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-26T04:13:28.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-26T04:13:28.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.bloom_locality: 0 2024-03-26T04:13:28.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.max_successive_merges: 0 2024-03-26T04:13:28.941 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-26T04:13:28.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.paranoid_file_checks: 0 2024-03-26T04:13:28.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.force_consistency_checks: 1 2024-03-26T04:13:28.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.report_bg_io_stats: 0 2024-03-26T04:13:28.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.ttl: 2592000 2024-03-26T04:13:28.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-26T04:13:28.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-26T04:13:28.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-26T04:13:28.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.enable_blob_files: false 2024-03-26T04:13:28.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.min_blob_size: 0 2024-03-26T04:13:28.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.blob_file_size: 268435456 2024-03-26T04:13:28.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-26T04:13:28.942 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-26T04:13:28.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-26T04:13:28.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-26T04:13:28.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-26T04:13:28.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.blob_file_starting_level: 0 2024-03-26T04:13:28.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-26T04:13:28.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-26T04:13:28.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-26T04:13:28.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 26f9a9b5-641e-486f-9efc-a4c0a07ea747 2024-03-26T04:13:28.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711426408620932, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-26T04:13:28.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-26T04:13:28.943 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711426408621316, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1677, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 578, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 456, "raw_average_value_size": 91, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711426408, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "26f9a9b5-641e-486f-9efc-a4c0a07ea747", "db_session_id": "2PSUGTFMF5262059B06B", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-26T04:13:28.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711426408621397, "job": 1, "event": "recovery_finished"} 2024-03-26T04:13:28.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-26T04:13:28.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-26T04:13:28.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5570ba0aa000 2024-03-26T04:13:28.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: DB pointer 0x5570ba094000 2024-03-26T04:13:28.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-26T04:13:28.944 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-26T04:13:28.944 INFO:journalctl@ceph.mon.c.smithi016.stdout: ** DB Stats ** 2024-03-26T04:13:28.944 INFO:journalctl@ceph.mon.c.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T04:13:28.944 INFO:journalctl@ceph.mon.c.smithi016.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-26T04:13:28.944 INFO:journalctl@ceph.mon.c.smithi016.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T04:13:28.945 INFO:journalctl@ceph.mon.c.smithi016.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T04:13:28.945 INFO:journalctl@ceph.mon.c.smithi016.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-26T04:13:28.945 INFO:journalctl@ceph.mon.c.smithi016.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-26T04:13:28.945 INFO:journalctl@ceph.mon.c.smithi016.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-26T04:13:28.945 INFO:journalctl@ceph.mon.c.smithi016.stdout: 2024-03-26T04:13:28.945 INFO:journalctl@ceph.mon.c.smithi016.stdout: ** Compaction Stats [default] ** 2024-03-26T04:13:28.945 INFO:journalctl@ceph.mon.c.smithi016.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T04:13:28.945 INFO:journalctl@ceph.mon.c.smithi016.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-26T04:13:28.945 INFO:journalctl@ceph.mon.c.smithi016.stdout: L0 1/0 1.64 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T04:13:28.945 INFO:journalctl@ceph.mon.c.smithi016.stdout: Sum 1/0 1.64 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T04:13:28.945 INFO:journalctl@ceph.mon.c.smithi016.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T04:13:28.945 INFO:journalctl@ceph.mon.c.smithi016.stdout: 2024-03-26T04:13:28.946 INFO:journalctl@ceph.mon.c.smithi016.stdout: ** Compaction Stats [default] ** 2024-03-26T04:13:28.946 INFO:journalctl@ceph.mon.c.smithi016.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-26T04:13:28.946 INFO:journalctl@ceph.mon.c.smithi016.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-26T04:13:28.946 INFO:journalctl@ceph.mon.c.smithi016.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-26T04:13:28.946 INFO:journalctl@ceph.mon.c.smithi016.stdout: 2024-03-26T04:13:28.946 INFO:journalctl@ceph.mon.c.smithi016.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-26T04:13:28.946 INFO:journalctl@ceph.mon.c.smithi016.stdout: 2024-03-26T04:13:28.946 INFO:journalctl@ceph.mon.c.smithi016.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-26T04:13:28.946 INFO:journalctl@ceph.mon.c.smithi016.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-26T04:13:28.946 INFO:journalctl@ceph.mon.c.smithi016.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-26T04:13:28.946 INFO:journalctl@ceph.mon.c.smithi016.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-26T04:13:28.947 INFO:journalctl@ceph.mon.c.smithi016.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-26T04:13:28.947 INFO:journalctl@ceph.mon.c.smithi016.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-26T04:13:28.947 INFO:journalctl@ceph.mon.c.smithi016.stdout: Cumulative compaction: 0.00 GB write, 0.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T04:13:28.947 INFO:journalctl@ceph.mon.c.smithi016.stdout: Interval compaction: 0.00 GB write, 0.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-26T04:13:28.947 INFO:journalctl@ceph.mon.c.smithi016.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-26T04:13:28.947 INFO:journalctl@ceph.mon.c.smithi016.stdout: Block cache BinnedLRUCache@0x5570b9fb9090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-26T04:13:28.947 INFO:journalctl@ceph.mon.c.smithi016.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-26T04:13:28.947 INFO:journalctl@ceph.mon.c.smithi016.stdout: 2024-03-26T04:13:28.947 INFO:journalctl@ceph.mon.c.smithi016.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-26T04:13:28.947 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-26T04:13:28.947 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: using public_addrv v1:172.21.15.16:6790/0 2024-03-26T04:13:28.948 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: starting mon.c rank -1 at public addrs v1:172.21.15.16:6790/0 at bind addrs v1:172.21.15.16:6790/0 mon_data /var/lib/ceph/mon/ceph-c fsid d7d59826-eb26-11ee-95d0-87774f69a715 2024-03-26T04:13:28.948 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: mon.c@-1(???) e0 preinit fsid d7d59826-eb26-11ee-95d0-87774f69a715 2024-03-26T04:13:28.948 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: mon.c@-1(synchronizing).mds e1 new map 2024-03-26T04:13:28.948 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-26T04:13:28.948 INFO:journalctl@ceph.mon.c.smithi016.stdout: e1 2024-03-26T04:13:28.948 INFO:journalctl@ceph.mon.c.smithi016.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-26T04:13:28.948 INFO:journalctl@ceph.mon.c.smithi016.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-26T04:13:28.948 INFO:journalctl@ceph.mon.c.smithi016.stdout: legacy client fscid: -1 2024-03-26T04:13:28.948 INFO:journalctl@ceph.mon.c.smithi016.stdout: 2024-03-26T04:13:28.948 INFO:journalctl@ceph.mon.c.smithi016.stdout: No filesystems configured 2024-03-26T04:13:28.948 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-26T04:13:28.948 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-26T04:13:28.949 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-26T04:13:28.949 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-26T04:13:28.949 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-26T04:13:28.949 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-26T04:13:28.949 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-26T04:13:28.949 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T04:13:28.949 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T04:13:28.949 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-26T04:13:28.949 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:28.949 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:28.949 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: from='client.14180 v1:172.21.15.16:0/3695379932' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:13:28.950 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:28.950 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1934386500' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-26T04:13:28.950 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:28.950 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:28.950 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:28.950 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:28.950 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi183", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:13:28.950 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:28.950 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:13:28.950 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: Updating smithi183:/etc/ceph/ceph.conf 2024-03-26T04:13:28.950 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: Updating smithi183:/var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:13:28.950 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:28.951 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: Updating smithi183:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:13:28.951 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1934386500' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-26T04:13:28.951 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:13:28.951 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: Updating smithi183:/var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:13:28.951 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:28.951 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:28.951 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:28.951 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:28.951 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: from='client.14184 v1:172.21.15.183:0/3681984561' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi016:[v1:172.21.15.16:6789]=a;smithi016:[v1:172.21.15.16:6790]=c;smithi183:[v1:172.21.15.183:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:13:28.951 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: Saving service mon spec with placement smithi016:[v1:172.21.15.16:6789]=a;smithi016:[v1:172.21.15.16:6790]=c;smithi183:[v1:172.21.15.183:6789]=b;count:3 2024-03-26T04:13:28.951 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:28.951 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:13:28.952 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:28.952 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:13:28.952 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:28.952 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:13:28.952 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:28.952 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: Deploying daemon mon.b on smithi183 2024-03-26T04:13:28.952 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:28.952 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:28.952 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.183:0/827647761' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T04:13:28.952 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:28.952 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:13:28.953 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: mon.a calling monitor election 2024-03-26T04:13:28.953 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:13:28.953 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:13:28.953 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:28.953 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:13:28.953 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: mon.b calling monitor election 2024-03-26T04:13:28.953 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:13:28.953 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:28.953 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:13:28.953 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:13:28.953 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-26T04:13:28.953 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: monmap e2: 2 mons at {a=v1:172.21.15.16:6789/0,b=v1:172.21.15.183:6789/0} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:13:28.954 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: fsmap 2024-03-26T04:13:28.954 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:13:28.954 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: mgrmap e13: y(active, since 41s) 2024-03-26T04:13:28.954 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: overall HEALTH_OK 2024-03-26T04:13:28.954 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:28.954 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:28.954 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:28.954 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:13:28.954 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:28.954 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: Deploying daemon mon.c on smithi016 2024-03-26T04:13:28.954 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.183:0/1141374543' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T04:13:28.955 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:13:28.955 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:28.955 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 ceph-mon[37215]: mon.c@-1(synchronizing).paxosservice(auth 1..4) refresh upgraded, format 0 -> 3 2024-03-26T04:13:28.955 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 bash[37196]: 2d20f53bd9c70a6c1116fe28f7ff96f3ff67dc8a3984598780567611f352f49c 2024-03-26T04:13:28.955 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:28 smithi016 systemd[1]: Started Ceph mon.c for d7d59826-eb26-11ee-95d0-87774f69a715. 2024-03-26T04:13:33.689 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:33.688+0000 7f3c30e78700 1 -- 172.21.15.183:0/56969585 >> v1:172.21.15.16:6789/0 conn(0x7f3c0c0035f0 legacy=0x7f3c0c005a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:13:33.689 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:33.688+0000 7f3c30e78700 1 -- 172.21.15.183:0/56969585 shutdown_connections 2024-03-26T04:13:33.689 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:33.688+0000 7f3c30e78700 1 -- 172.21.15.183:0/56969585 >> 172.21.15.183:0/56969585 conn(0x7f3c2c0fd5c0 msgr2=0x7f3c2c0ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:13:33.689 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:33.688+0000 7f3c30e78700 1 -- 172.21.15.183:0/56969585 shutdown_connections 2024-03-26T04:13:33.689 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:33.688+0000 7f3c30e78700 1 -- 172.21.15.183:0/56969585 wait complete. 2024-03-26T04:13:33.690 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:33.689+0000 7f3c30e78700 1 Processor -- start 2024-03-26T04:13:33.690 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:33.689+0000 7f3c30e78700 1 -- start start 2024-03-26T04:13:33.690 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:33.690+0000 7f3c30e78700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3c2c1acc70 con 0x7f3c2c19d420 2024-03-26T04:13:33.690 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:33.690+0000 7f3c30e78700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3c2c1ade50 con 0x7f3c2c1a8fa0 2024-03-26T04:13:33.691 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:33.690+0000 7f3c30e78700 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3c2c1af050 con 0x7f3c0c0035f0 2024-03-26T04:13:33.691 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:33.690+0000 7f3c2ad9d700 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f3c2c1a8fa0 0x7f3c2c1ab390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:38792/0 (socket says 172.21.15.183:38792) 2024-03-26T04:13:33.691 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:33.690+0000 7f3c2ad9d700 1 -- 172.21.15.183:0/3927306148 learned_addr learned my addr 172.21.15.183:0/3927306148 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-26T04:13:33.692 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:33.691+0000 7f3c23fff700 1 -- 172.21.15.183:0/3927306148 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3793671423 0 0) 0x7f3c2c1acc70 con 0x7f3c2c19d420 2024-03-26T04:13:33.692 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:33.692+0000 7f3c23fff700 1 -- 172.21.15.183:0/3927306148 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c18003bb0 con 0x7f3c2c19d420 2024-03-26T04:13:33.692 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:33.692+0000 7f3c23fff700 1 -- 172.21.15.183:0/3927306148 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 306034594 0 0) 0x7f3c18003bb0 con 0x7f3c2c19d420 2024-03-26T04:13:33.693 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:33.692+0000 7f3c23fff700 1 -- 172.21.15.183:0/3927306148 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c2c1acc70 con 0x7f3c2c19d420 2024-03-26T04:13:33.693 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:33.692+0000 7f3c23fff700 1 -- 172.21.15.183:0/3927306148 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f3c14001830 con 0x7f3c2c19d420 2024-03-26T04:13:33.694 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:33.692+0000 7f3c23fff700 1 -- 172.21.15.183:0/3927306148 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1832378893 0 0) 0x7f3c2c1acc70 con 0x7f3c2c19d420 2024-03-26T04:13:33.694 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:33.693+0000 7f3c23fff700 1 -- 172.21.15.183:0/3927306148 >> v1:172.21.15.16:6790/0 conn(0x7f3c0c0035f0 legacy=0x7f3c2c19cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:13:33.694 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:33.693+0000 7f3c23fff700 1 -- 172.21.15.183:0/3927306148 >> v1:172.21.15.183:6789/0 conn(0x7f3c2c1a8fa0 legacy=0x7f3c2c1ab390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:13:33.694 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:33.693+0000 7f3c23fff700 1 -- 172.21.15.183:0/3927306148 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c2c1b0250 con 0x7f3c2c19d420 2024-03-26T04:13:33.694 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:33.693+0000 7f3c23fff700 1 -- 172.21.15.183:0/3927306148 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f3c14003b10 con 0x7f3c2c19d420 2024-03-26T04:13:33.694 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:33.693+0000 7f3c30e78700 1 -- 172.21.15.183:0/3927306148 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c2c1ace60 con 0x7f3c2c19d420 2024-03-26T04:13:33.694 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:33.693+0000 7f3c23fff700 1 -- 172.21.15.183:0/3927306148 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f3c14004a50 con 0x7f3c2c19d420 2024-03-26T04:13:33.694 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:33.693+0000 7f3c30e78700 1 -- 172.21.15.183:0/3927306148 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c2c1ad320 con 0x7f3c2c19d420 2024-03-26T04:13:33.695 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:33.694+0000 7f3c23fff700 1 -- 172.21.15.183:0/3927306148 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 13) v1 ==== 45949+0+0 (unknown 1805086598 0 0) 0x7f3c14005d80 con 0x7f3c2c19d420 2024-03-26T04:13:33.695 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:33.694+0000 7f3c30e78700 1 -- 172.21.15.183:0/3927306148 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c2c0636b0 con 0x7f3c2c19d420 2024-03-26T04:13:33.695 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:33.694+0000 7f3c23fff700 1 -- 172.21.15.183:0/3927306148 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 1069+0+0 (unknown 21079357 0 0) 0x7f3c14046010 con 0x7f3c2c19d420 2024-03-26T04:13:33.701 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:33.700+0000 7f3c23fff700 1 -- 172.21.15.183:0/3927306148 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f3c14006690 con 0x7f3c2c19d420 2024-03-26T04:13:33.995 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:33.995+0000 7f3c30e78700 1 -- 172.21.15.183:0/3927306148 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3c2c19e070 con 0x7f3c2c19d420 2024-03-26T04:13:33.996 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:33 smithi183 ceph-mon[33745]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:33.996 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:33 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:13:33.996 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:33 smithi183 ceph-mon[33745]: mon.a calling monitor election 2024-03-26T04:13:33.996 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:33 smithi183 ceph-mon[33745]: mon.b calling monitor election 2024-03-26T04:13:33.996 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:33 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:13:33.996 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:33 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:13:33.996 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:33 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:13:33.997 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:33 smithi183 ceph-mon[33745]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:33.997 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:33 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:13:33.997 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:33 smithi183 ceph-mon[33745]: mon.c calling monitor election 2024-03-26T04:13:33.997 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:33 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:13:33.997 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:33 smithi183 ceph-mon[33745]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:33.997 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:33 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:13:33.997 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:33 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:13:33.997 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:33 smithi183 ceph-mon[33745]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-26T04:13:33.997 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:33 smithi183 ceph-mon[33745]: monmap e3: 3 mons at {a=v1:172.21.15.16:6789/0,b=v1:172.21.15.183:6789/0,c=v1:172.21.15.16:6790/0} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:13:33.997 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:33 smithi183 ceph-mon[33745]: fsmap 2024-03-26T04:13:33.997 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:33 smithi183 ceph-mon[33745]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:13:33.998 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:33 smithi183 ceph-mon[33745]: mgrmap e13: y(active, since 49s) 2024-03-26T04:13:33.998 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:33 smithi183 ceph-mon[33745]: overall HEALTH_OK 2024-03-26T04:13:33.998 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:33 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:33.998 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:33 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:33.998 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:33 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:33.998 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:33 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:33.998 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:33.995+0000 7f3c23fff700 1 -- 172.21.15.183:0/3927306148 <== mon.0 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1127 (unknown 409899127 0 3067420119) 0x7f3c14015430 con 0x7f3c2c19d420 2024-03-26T04:13:33.999 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-26T04:13:33.999 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":3,"fsid":"d7d59826-eb26-11ee-95d0-87774f69a715","modified":"2024-03-26T04:13:28.639648Z","created":"2024-03-26T04:11:44.857505Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6789","nonce":0}]},"addr":"172.21.15.16:6789/0","public_addr":"172.21.15.16:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6789","nonce":0}]},"addr":"172.21.15.183:6789/0","public_addr":"172.21.15.183:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6790","nonce":0}]},"addr":"172.21.15.16:6790/0","public_addr":"172.21.15.16:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-26T04:13:34.000 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:33.997+0000 7f3c30e78700 1 -- 172.21.15.183:0/3927306148 >> v1:172.21.15.16:6800/4042489921 conn(0x7f3c1803a140 legacy=0x7f3c1803c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:13:34.000 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:33.997+0000 7f3c30e78700 1 -- 172.21.15.183:0/3927306148 >> v1:172.21.15.16:6789/0 conn(0x7f3c2c19d420 legacy=0x7f3c2c1a7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:13:34.000 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:33.997+0000 7f3c30e78700 1 -- 172.21.15.183:0/3927306148 shutdown_connections 2024-03-26T04:13:34.000 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:33.997+0000 7f3c30e78700 1 -- 172.21.15.183:0/3927306148 >> 172.21.15.183:0/3927306148 conn(0x7f3c2c0fd5c0 msgr2=0x7f3c2c1050c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:13:34.000 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:33.998+0000 7f3c30e78700 1 -- 172.21.15.183:0/3927306148 shutdown_connections 2024-03-26T04:13:34.000 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:33.998+0000 7f3c30e78700 1 -- 172.21.15.183:0/3927306148 wait complete. 2024-03-26T04:13:34.000 INFO:teuthology.orchestra.run.smithi183.stderr:dumped monmap epoch 3 2024-03-26T04:13:34.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[29101]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:34.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:13:34.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[29101]: mon.a calling monitor election 2024-03-26T04:13:34.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[29101]: mon.b calling monitor election 2024-03-26T04:13:34.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:13:34.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:13:34.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:13:34.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[29101]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:34.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:13:34.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[29101]: mon.c calling monitor election 2024-03-26T04:13:34.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:13:34.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[29101]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:34.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:13:34.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:13:34.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[29101]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-26T04:13:34.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[29101]: monmap e3: 3 mons at {a=v1:172.21.15.16:6789/0,b=v1:172.21.15.183:6789/0,c=v1:172.21.15.16:6790/0} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:13:34.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[29101]: fsmap 2024-03-26T04:13:34.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[29101]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:13:34.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[29101]: mgrmap e13: y(active, since 49s) 2024-03-26T04:13:34.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[29101]: overall HEALTH_OK 2024-03-26T04:13:34.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:34.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:34.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:34.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:34.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[37215]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:34.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:13:34.165 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[37215]: mon.a calling monitor election 2024-03-26T04:13:34.165 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[37215]: mon.b calling monitor election 2024-03-26T04:13:34.165 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:13:34.165 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:13:34.165 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:13:34.165 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[37215]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:34.165 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:13:34.165 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[37215]: mon.c calling monitor election 2024-03-26T04:13:34.165 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:13:34.165 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[37215]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:34.165 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:13:34.165 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:13:34.165 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[37215]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-26T04:13:34.166 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[37215]: monmap e3: 3 mons at {a=v1:172.21.15.16:6789/0,b=v1:172.21.15.183:6789/0,c=v1:172.21.15.16:6790/0} removed_ranks: {} disallowed_leaders: {} 2024-03-26T04:13:34.166 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[37215]: fsmap 2024-03-26T04:13:34.166 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[37215]: osdmap e4: 0 total, 0 up, 0 in 2024-03-26T04:13:34.166 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[37215]: mgrmap e13: y(active, since 49s) 2024-03-26T04:13:34.166 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[37215]: overall HEALTH_OK 2024-03-26T04:13:34.166 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:34.166 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:34.166 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:34.166 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:33 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:34.592 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-26T04:13:34.593 DEBUG:teuthology.orchestra.run.smithi016:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph config generate-minimal-conf 2024-03-26T04:13:34.855 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:34 smithi183 ceph-mon[33745]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:34.855 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:34 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:34.855 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:34 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:13:34.855 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:34 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.183:0/3927306148' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T04:13:34.855 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:34 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:13:34.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:34 smithi016 ceph-mon[29101]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:34.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:34 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:34.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:34 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:13:34.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:34 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.183:0/3927306148' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T04:13:34.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:34 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:13:34.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:34 smithi016 ceph-mon[37215]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:34.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:34 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:34.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:34 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:13:34.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:34 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.183:0/3927306148' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-26T04:13:34.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:34 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:13:34.973 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:13:36.143 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:36 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:36.143 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:36 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:36.143 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:36 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:36.143 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:36 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:36.143 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:36 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:36.143 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:36 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:13:36.143 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:36 smithi016 ceph-mon[37215]: Updating smithi016:/etc/ceph/ceph.conf 2024-03-26T04:13:36.143 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:36 smithi016 ceph-mon[37215]: Updating smithi183:/etc/ceph/ceph.conf 2024-03-26T04:13:36.143 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:36 smithi016 ceph-mon[37215]: Updating smithi183:/var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:13:36.143 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:36 smithi016 ceph-mon[37215]: Updating smithi016:/var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:13:36.144 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:36 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:36.144 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:36 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:36.144 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:36 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:36.144 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:36 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:36.144 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:36 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:36.144 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:36 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:36.144 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:36 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:36.144 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:36 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:36.144 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:36 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:36.144 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:36 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:13:36.144 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:36 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:13:36.144 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:36 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:36.144 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:36 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:36.145 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:36 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:36.145 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:36 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:36.145 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:36 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:36.145 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:36 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:36.145 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:36 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:13:36.145 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:36 smithi016 ceph-mon[29101]: Updating smithi016:/etc/ceph/ceph.conf 2024-03-26T04:13:36.145 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:36 smithi016 ceph-mon[29101]: Updating smithi183:/etc/ceph/ceph.conf 2024-03-26T04:13:36.145 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:36 smithi016 ceph-mon[29101]: Updating smithi183:/var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:13:36.145 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:36 smithi016 ceph-mon[29101]: Updating smithi016:/var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:13:36.145 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:36 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:36.145 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:36 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:36.145 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:36 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:36.145 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:36 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:36.145 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:36 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:36.145 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:36 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:36.146 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:36 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:36.146 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:36 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:36.146 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:36 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:36.146 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:36 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:13:36.146 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:36 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:13:36.146 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:36 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:36.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:36 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:36.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:36 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:36.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:36 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:36.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:36 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:36.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:36 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:36.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:36 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:13:36.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:36 smithi183 ceph-mon[33745]: Updating smithi016:/etc/ceph/ceph.conf 2024-03-26T04:13:36.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:36 smithi183 ceph-mon[33745]: Updating smithi183:/etc/ceph/ceph.conf 2024-03-26T04:13:36.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:36 smithi183 ceph-mon[33745]: Updating smithi183:/var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:13:36.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:36 smithi183 ceph-mon[33745]: Updating smithi016:/var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:13:36.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:36 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:36.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:36 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:36.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:36 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:36.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:36 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:36.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:36 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:36.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:36 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:36.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:36 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:36.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:36 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:36.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:36 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:36.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:36 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:13:36.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:36 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:13:36.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:36 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:36.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:36.631+0000 7f917d607700 1 -- 172.21.15.16:0/1469691216 >> v1:172.21.15.16:6789/0 conn(0x7f91781068a0 legacy=0x7f9178106c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:13:36.633 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:36.631+0000 7f917d607700 1 -- 172.21.15.16:0/1469691216 shutdown_connections 2024-03-26T04:13:36.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:36.631+0000 7f917d607700 1 -- 172.21.15.16:0/1469691216 >> 172.21.15.16:0/1469691216 conn(0x7f91780763b0 msgr2=0x7f91780767c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:13:36.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:36.631+0000 7f917d607700 1 -- 172.21.15.16:0/1469691216 shutdown_connections 2024-03-26T04:13:36.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:36.631+0000 7f917d607700 1 -- 172.21.15.16:0/1469691216 wait complete. 2024-03-26T04:13:36.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:36.632+0000 7f917d607700 1 Processor -- start 2024-03-26T04:13:36.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:36.632+0000 7f917d607700 1 -- start start 2024-03-26T04:13:36.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:36.632+0000 7f917d607700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f91781a6750 con 0x7f917819cb70 2024-03-26T04:13:36.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:36.632+0000 7f917d607700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f91781a7950 con 0x7f91781a2a40 2024-03-26T04:13:36.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:36.632+0000 7f917d607700 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f91781a8b50 con 0x7f91781068a0 2024-03-26T04:13:36.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:36.632+0000 7f91767fc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f917819cb70 0x7f91781a1320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38346/0 (socket says 172.21.15.16:38346) 2024-03-26T04:13:36.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:36.632+0000 7f91767fc700 1 -- 172.21.15.16:0/1129735757 learned_addr learned my addr 172.21.15.16:0/1129735757 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:13:36.634 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:36.632+0000 7f9176ffd700 1 --1- 172.21.15.16:0/1129735757 >> v1:172.21.15.16:6790/0 conn(0x7f91781068a0 0x7f917819c460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6790/0 says I am v1:172.21.15.16:48968/0 (socket says 172.21.15.16:48968) 2024-03-26T04:13:36.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:36.632+0000 7f916ffff700 1 -- 172.21.15.16:0/1129735757 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3058694346 0 0) 0x7f91781a6750 con 0x7f917819cb70 2024-03-26T04:13:36.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:36.632+0000 7f916ffff700 1 -- 172.21.15.16:0/1129735757 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9164003c00 con 0x7f917819cb70 2024-03-26T04:13:36.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:36.632+0000 7f916ffff700 1 -- 172.21.15.16:0/1129735757 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2009265895 0 0) 0x7f91781a8b50 con 0x7f91781068a0 2024-03-26T04:13:36.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:36.632+0000 7f916ffff700 1 -- 172.21.15.16:0/1129735757 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91781a6750 con 0x7f91781068a0 2024-03-26T04:13:36.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:36.633+0000 7f916ffff700 1 -- 172.21.15.16:0/1129735757 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2694010205 0 0) 0x7f91781a7950 con 0x7f91781a2a40 2024-03-26T04:13:36.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:36.633+0000 7f916ffff700 1 -- 172.21.15.16:0/1129735757 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91781a8b50 con 0x7f91781a2a40 2024-03-26T04:13:36.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:36.633+0000 7f916ffff700 1 -- 172.21.15.16:0/1129735757 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 81784798 0 0) 0x7f9164003c00 con 0x7f917819cb70 2024-03-26T04:13:36.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:36.633+0000 7f916ffff700 1 -- 172.21.15.16:0/1129735757 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91781a7950 con 0x7f917819cb70 2024-03-26T04:13:36.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:36.633+0000 7f916ffff700 1 -- 172.21.15.16:0/1129735757 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f9168001760 con 0x7f917819cb70 2024-03-26T04:13:36.635 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:36.633+0000 7f916ffff700 1 -- 172.21.15.16:0/1129735757 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4248888738 0 0) 0x7f91781a6750 con 0x7f91781068a0 2024-03-26T04:13:36.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:36.633+0000 7f916ffff700 1 -- 172.21.15.16:0/1129735757 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9164003c00 con 0x7f91781068a0 2024-03-26T04:13:36.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:36.633+0000 7f916ffff700 1 -- 172.21.15.16:0/1129735757 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f91600045e0 con 0x7f91781068a0 2024-03-26T04:13:36.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:36.633+0000 7f916ffff700 1 -- 172.21.15.16:0/1129735757 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1854767032 0 0) 0x7f91781a7950 con 0x7f917819cb70 2024-03-26T04:13:36.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:36.633+0000 7f916ffff700 1 -- 172.21.15.16:0/1129735757 >> v1:172.21.15.16:6790/0 conn(0x7f91781068a0 legacy=0x7f917819c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:13:36.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:36.633+0000 7f916ffff700 1 -- 172.21.15.16:0/1129735757 >> v1:172.21.15.183:6789/0 conn(0x7f91781a2a40 legacy=0x7f91781a4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:13:36.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:36.633+0000 7f916ffff700 1 -- 172.21.15.16:0/1129735757 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91781a9d50 con 0x7f917819cb70 2024-03-26T04:13:36.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:36.633+0000 7f917d607700 1 -- 172.21.15.16:0/1129735757 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91781a6940 con 0x7f917819cb70 2024-03-26T04:13:36.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:36.633+0000 7f917d607700 1 -- 172.21.15.16:0/1129735757 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f91781a6dc0 con 0x7f917819cb70 2024-03-26T04:13:36.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:36.633+0000 7f916ffff700 1 -- 172.21.15.16:0/1129735757 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f9168004dc0 con 0x7f917819cb70 2024-03-26T04:13:36.636 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:36.633+0000 7f916ffff700 1 -- 172.21.15.16:0/1129735757 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f9168005c20 con 0x7f917819cb70 2024-03-26T04:13:36.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:36.634+0000 7f917d607700 1 -- 172.21.15.16:0/1129735757 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9158005120 con 0x7f917819cb70 2024-03-26T04:13:36.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:36.634+0000 7f916ffff700 1 -- 172.21.15.16:0/1129735757 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 13) v1 ==== 45949+0+0 (unknown 1805086598 0 0) 0x7f9168005eb0 con 0x7f917819cb70 2024-03-26T04:13:36.637 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:36.634+0000 7f916ffff700 1 -- 172.21.15.16:0/1129735757 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 1069+0+0 (unknown 21079357 0 0) 0x7f9168046e30 con 0x7f917819cb70 2024-03-26T04:13:36.639 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:36.639+0000 7f916ffff700 1 -- 172.21.15.16:0/1129735757 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f9168006810 con 0x7f917819cb70 2024-03-26T04:13:36.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:36.847+0000 7f917d607700 1 -- 172.21.15.16:0/1129735757 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f91580053d0 con 0x7f917819cb70 2024-03-26T04:13:36.848 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:36.848+0000 7f916ffff700 1 -- 172.21.15.16:0/1129735757 <== mon.0 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+191 (unknown 2189217731 0 3237390104) 0x7f9168016360 con 0x7f917819cb70 2024-03-26T04:13:36.848 INFO:teuthology.orchestra.run.smithi016.stdout:# minimal ceph.conf for d7d59826-eb26-11ee-95d0-87774f69a715 2024-03-26T04:13:36.848 INFO:teuthology.orchestra.run.smithi016.stdout:[global] 2024-03-26T04:13:36.848 INFO:teuthology.orchestra.run.smithi016.stdout: fsid = d7d59826-eb26-11ee-95d0-87774f69a715 2024-03-26T04:13:36.849 INFO:teuthology.orchestra.run.smithi016.stdout: mon_host = 172.21.15.16:6789/0 172.21.15.183:6789/0 v1:172.21.15.16:6790/0 2024-03-26T04:13:36.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:36.850+0000 7f917d607700 1 -- 172.21.15.16:0/1129735757 >> v1:172.21.15.16:6800/4042489921 conn(0x7f916403a1d0 legacy=0x7f916403c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:13:36.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:36.850+0000 7f917d607700 1 -- 172.21.15.16:0/1129735757 >> v1:172.21.15.16:6789/0 conn(0x7f917819cb70 legacy=0x7f91781a1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:13:36.850 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:36.850+0000 7f917d607700 1 -- 172.21.15.16:0/1129735757 shutdown_connections 2024-03-26T04:13:36.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:36.850+0000 7f917d607700 1 -- 172.21.15.16:0/1129735757 >> 172.21.15.16:0/1129735757 conn(0x7f91780763b0 msgr2=0x7f9178069600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:13:36.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:36.850+0000 7f917d607700 1 -- 172.21.15.16:0/1129735757 shutdown_connections 2024-03-26T04:13:36.851 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:36.850+0000 7f917d607700 1 -- 172.21.15.16:0/1129735757 wait complete. 2024-03-26T04:13:37.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:37 smithi016 ceph-mon[29101]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:37.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:37 smithi016 ceph-mon[29101]: Reconfiguring mon.a (unknown last config time)... 2024-03-26T04:13:37.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:37 smithi016 ceph-mon[29101]: Reconfiguring daemon mon.a on smithi016 2024-03-26T04:13:37.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:37 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1129735757' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:37.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:37 smithi016 ceph-mon[37215]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:37.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:37 smithi016 ceph-mon[37215]: Reconfiguring mon.a (unknown last config time)... 2024-03-26T04:13:37.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:37 smithi016 ceph-mon[37215]: Reconfiguring daemon mon.a on smithi016 2024-03-26T04:13:37.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:37 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1129735757' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:37.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:37 smithi183 ceph-mon[33745]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:37.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:37 smithi183 ceph-mon[33745]: Reconfiguring mon.a (unknown last config time)... 2024-03-26T04:13:37.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:37 smithi183 ceph-mon[33745]: Reconfiguring daemon mon.a on smithi016 2024-03-26T04:13:37.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:37 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1129735757' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:37.836 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-26T04:13:37.836 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-26T04:13:37.836 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.conf 2024-03-26T04:13:37.869 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-26T04:13:37.869 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:13:37.938 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-26T04:13:37.938 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/ceph/ceph.conf 2024-03-26T04:13:37.972 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-26T04:13:37.973 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:13:38.045 INFO:tasks.cephadm:Adding mgr.y on smithi016 2024-03-26T04:13:38.045 INFO:tasks.cephadm:Adding mgr.x on smithi183 2024-03-26T04:13:38.045 DEBUG:teuthology.orchestra.run.smithi183:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph orch apply mgr '2;smithi016=y;smithi183=x' 2024-03-26T04:13:38.319 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:13:39.614 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:39 smithi183 ceph-mon[33745]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:39.614 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:39 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:39.614 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:39 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:39.614 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:39 smithi183 ceph-mon[33745]: Reconfiguring mon.c (monmap changed)... 2024-03-26T04:13:39.614 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:39 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:13:39.614 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:39 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:13:39.614 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:39 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:39.614 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:39 smithi183 ceph-mon[33745]: Reconfiguring daemon mon.c on smithi016 2024-03-26T04:13:39.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:39 smithi016 ceph-mon[29101]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:39.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:39 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:39.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:39 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:39.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:39 smithi016 ceph-mon[29101]: Reconfiguring mon.c (monmap changed)... 2024-03-26T04:13:39.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:39 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:13:39.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:39 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:13:39.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:39 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:39.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:39 smithi016 ceph-mon[29101]: Reconfiguring daemon mon.c on smithi016 2024-03-26T04:13:39.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:39 smithi016 ceph-mon[37215]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:39.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:39 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:39.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:39 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:39.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:39 smithi016 ceph-mon[37215]: Reconfiguring mon.c (monmap changed)... 2024-03-26T04:13:39.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:39 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:13:39.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:39 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:13:39.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:39 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:39.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:39 smithi016 ceph-mon[37215]: Reconfiguring daemon mon.c on smithi016 2024-03-26T04:13:39.845 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:39.844+0000 7f22343dc700 1 -- 172.21.15.183:0/1851496691 >> v1:172.21.15.16:6789/0 conn(0x7f222c101b30 legacy=0x7f222c101f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:13:39.845 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:39.844+0000 7f22343dc700 1 -- 172.21.15.183:0/1851496691 shutdown_connections 2024-03-26T04:13:39.845 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:39.844+0000 7f22343dc700 1 -- 172.21.15.183:0/1851496691 >> 172.21.15.183:0/1851496691 conn(0x7f222c0fd0d0 msgr2=0x7f222c0ff510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:13:39.845 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:39.844+0000 7f22343dc700 1 -- 172.21.15.183:0/1851496691 shutdown_connections 2024-03-26T04:13:39.846 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:39.844+0000 7f22343dc700 1 -- 172.21.15.183:0/1851496691 wait complete. 2024-03-26T04:13:39.846 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:39.845+0000 7f22343dc700 1 Processor -- start 2024-03-26T04:13:39.846 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:39.845+0000 7f22343dc700 1 -- start start 2024-03-26T04:13:39.846 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:39.845+0000 7f22343dc700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f222c1a3ec0 con 0x7f222c1a0250 2024-03-26T04:13:39.846 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:39.845+0000 7f22343dc700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f222c1a50a0 con 0x7f222c101b30 2024-03-26T04:13:39.846 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:39.845+0000 7f22343dc700 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f222c1a62a0 con 0x7f222c198d60 2024-03-26T04:13:39.846 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:39.845+0000 7f2232979700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f222c1a0250 0x7f222c1a2640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.183:59484/0 (socket says 172.21.15.183:59484) 2024-03-26T04:13:39.846 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:39.845+0000 7f2232979700 1 -- 172.21.15.183:0/1029239341 learned_addr learned my addr 172.21.15.183:0/1029239341 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-26T04:13:39.848 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:39.845+0000 7f222b7fe700 1 -- 172.21.15.183:0/1029239341 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1006254158 0 0) 0x7f222c1a50a0 con 0x7f222c101b30 2024-03-26T04:13:39.848 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:39.845+0000 7f222b7fe700 1 -- 172.21.15.183:0/1029239341 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2214003bb0 con 0x7f222c101b30 2024-03-26T04:13:39.848 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:39.845+0000 7f222b7fe700 1 -- 172.21.15.183:0/1029239341 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1215154581 0 0) 0x7f222c1a3ec0 con 0x7f222c1a0250 2024-03-26T04:13:39.848 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:39.845+0000 7f222b7fe700 1 -- 172.21.15.183:0/1029239341 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f222c1a50a0 con 0x7f222c1a0250 2024-03-26T04:13:39.848 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:39.845+0000 7f222b7fe700 1 -- 172.21.15.183:0/1029239341 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1990434633 0 0) 0x7f222c1a62a0 con 0x7f222c198d60 2024-03-26T04:13:39.848 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:39.845+0000 7f222b7fe700 1 -- 172.21.15.183:0/1029239341 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f222c1a3ec0 con 0x7f222c198d60 2024-03-26T04:13:39.848 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:39.845+0000 7f222b7fe700 1 -- 172.21.15.183:0/1029239341 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 568886383 0 0) 0x7f2214003bb0 con 0x7f222c101b30 2024-03-26T04:13:39.848 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:39.845+0000 7f222b7fe700 1 -- 172.21.15.183:0/1029239341 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f222c1a62a0 con 0x7f222c101b30 2024-03-26T04:13:39.849 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:39.846+0000 7f222b7fe700 1 -- 172.21.15.183:0/1029239341 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f221c000fe0 con 0x7f222c101b30 2024-03-26T04:13:39.849 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:39.846+0000 7f222b7fe700 1 -- 172.21.15.183:0/1029239341 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 474259144 0 0) 0x7f222c1a3ec0 con 0x7f222c198d60 2024-03-26T04:13:39.849 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:39.846+0000 7f222b7fe700 1 -- 172.21.15.183:0/1029239341 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2214003bb0 con 0x7f222c198d60 2024-03-26T04:13:39.849 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:39.846+0000 7f222b7fe700 1 -- 172.21.15.183:0/1029239341 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3343971181 0 0) 0x7f222c1a50a0 con 0x7f222c1a0250 2024-03-26T04:13:39.849 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:39.846+0000 7f222b7fe700 1 -- 172.21.15.183:0/1029239341 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f222c1a3ec0 con 0x7f222c1a0250 2024-03-26T04:13:39.849 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:39.846+0000 7f222b7fe700 1 -- 172.21.15.183:0/1029239341 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f22200016d0 con 0x7f222c198d60 2024-03-26T04:13:39.849 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:39.846+0000 7f222b7fe700 1 -- 172.21.15.183:0/1029239341 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f22240019e0 con 0x7f222c1a0250 2024-03-26T04:13:39.849 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:39.846+0000 7f222b7fe700 1 -- 172.21.15.183:0/1029239341 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 210342094 0 0) 0x7f222c1a62a0 con 0x7f222c101b30 2024-03-26T04:13:39.849 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:39.846+0000 7f222b7fe700 1 -- 172.21.15.183:0/1029239341 >> v1:172.21.15.16:6790/0 conn(0x7f222c198d60 legacy=0x7f222c19eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:13:39.849 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:39.846+0000 7f222b7fe700 1 -- 172.21.15.183:0/1029239341 >> v1:172.21.15.16:6789/0 conn(0x7f222c1a0250 legacy=0x7f222c1a2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:13:39.849 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:39.846+0000 7f222b7fe700 1 -- 172.21.15.183:0/1029239341 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f222c1a74a0 con 0x7f222c101b30 2024-03-26T04:13:39.849 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:39.846+0000 7f22343dc700 1 -- 172.21.15.183:0/1029239341 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f222c1a40b0 con 0x7f222c101b30 2024-03-26T04:13:39.849 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:39.846+0000 7f22343dc700 1 -- 172.21.15.183:0/1029239341 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f222c1a45b0 con 0x7f222c101b30 2024-03-26T04:13:39.849 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:39.846+0000 7f222b7fe700 1 -- 172.21.15.183:0/1029239341 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f221c001bb0 con 0x7f222c101b30 2024-03-26T04:13:39.850 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:39.846+0000 7f222b7fe700 1 -- 172.21.15.183:0/1029239341 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f221c004d80 con 0x7f222c101b30 2024-03-26T04:13:39.850 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:39.847+0000 7f222b7fe700 1 -- 172.21.15.183:0/1029239341 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 13) v1 ==== 45949+0+0 (unknown 1805086598 0 0) 0x7f221c010270 con 0x7f222c101b30 2024-03-26T04:13:39.850 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:39.847+0000 7f22343dc700 1 -- 172.21.15.183:0/1029239341 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2218005120 con 0x7f222c101b30 2024-03-26T04:13:39.850 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:39.847+0000 7f222b7fe700 1 -- 172.21.15.183:0/1029239341 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 1069+0+0 (unknown 21079357 0 0) 0x7f221c046090 con 0x7f222c101b30 2024-03-26T04:13:39.852 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:39.852+0000 7f222b7fe700 1 -- 172.21.15.183:0/1029239341 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f221c011b40 con 0x7f222c101b30 2024-03-26T04:13:40.065 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:40.064+0000 7f22343dc700 1 -- 172.21.15.183:0/1029239341 --> v1:172.21.15.16:6800/4042489921 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi016=y;smithi183=x", "target": ["mon-mgr", ""]}) v1 -- 0x7f2218000b90 con 0x7f221403a6b0 2024-03-26T04:13:40.071 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:40.071+0000 7f222b7fe700 1 -- 172.21.15.183:0/1029239341 <== mgr.14152 v1:172.21.15.16:6800/4042489921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (unknown 0 0 325935098) 0x7f2218000b90 con 0x7f221403a6b0 2024-03-26T04:13:40.072 INFO:teuthology.orchestra.run.smithi183.stdout:Scheduled mgr update... 2024-03-26T04:13:40.074 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:40.073+0000 7f22343dc700 1 -- 172.21.15.183:0/1029239341 >> v1:172.21.15.16:6800/4042489921 conn(0x7f221403a6b0 legacy=0x7f221403cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:13:40.074 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:40.073+0000 7f22343dc700 1 -- 172.21.15.183:0/1029239341 >> v1:172.21.15.183:6789/0 conn(0x7f222c101b30 legacy=0x7f222c198650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:13:40.074 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:40.073+0000 7f22343dc700 1 -- 172.21.15.183:0/1029239341 shutdown_connections 2024-03-26T04:13:40.074 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:40.073+0000 7f22343dc700 1 -- 172.21.15.183:0/1029239341 >> 172.21.15.183:0/1029239341 conn(0x7f222c0fd0d0 msgr2=0x7f222c104700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:13:40.074 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:40.073+0000 7f22343dc700 1 -- 172.21.15.183:0/1029239341 shutdown_connections 2024-03-26T04:13:40.074 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:13:40.073+0000 7f22343dc700 1 -- 172.21.15.183:0/1029239341 wait complete. 2024-03-26T04:13:40.569 DEBUG:teuthology.orchestra.run.smithi183:mgr.x> sudo journalctl -f -n 0 -u ceph-d7d59826-eb26-11ee-95d0-87774f69a715@mgr.x.service 2024-03-26T04:13:40.572 INFO:tasks.cephadm:Deploying OSDs... 2024-03-26T04:13:40.572 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-26T04:13:40.572 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T04:13:40.591 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-26T04:13:40.591 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_1 2024-03-26T04:13:40.649 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-26T04:13:40.649 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:13:40.649 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-26T04:13:40.649 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:13:40.649 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:13:40.649 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-03-26 04:13:28.055457198 +0000 2024-03-26T04:13:40.649 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-03-26 04:03:45.858899581 +0000 2024-03-26T04:13:40.649 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-03-26 04:03:45.858899581 +0000 2024-03-26T04:13:40.650 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-03-26 04:03:45.858899581 +0000 2024-03-26T04:13:40.650 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-26T04:13:40.719 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-03-26T04:13:40.720 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-03-26T04:13:40.720 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000119929 s, 4.3 MB/s 2024-03-26T04:13:40.721 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-26T04:13:40.779 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_2 2024-03-26T04:13:40.837 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-26T04:13:40.837 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:13:40.837 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-26T04:13:40.837 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:13:40.837 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:13:40.837 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-03-26 04:13:28.056457182 +0000 2024-03-26T04:13:40.837 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-03-26 04:03:46.196893747 +0000 2024-03-26T04:13:40.837 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-03-26 04:03:46.196893747 +0000 2024-03-26T04:13:40.837 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-03-26 04:03:46.196893747 +0000 2024-03-26T04:13:40.838 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-26T04:13:40.905 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-03-26T04:13:40.905 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-03-26T04:13:40.905 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000100493 s, 5.1 MB/s 2024-03-26T04:13:40.906 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-26T04:13:40.964 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_3 2024-03-26T04:13:41.020 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:40 smithi016 ceph-mon[29101]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:41.020 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:40 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:41.020 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:40 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:41.020 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:40 smithi016 ceph-mon[29101]: Reconfiguring mon.b (monmap changed)... 2024-03-26T04:13:41.020 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:40 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:13:41.020 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:40 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:13:41.020 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:40 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:41.020 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:40 smithi016 ceph-mon[29101]: Reconfiguring daemon mon.b on smithi183 2024-03-26T04:13:41.020 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:40 smithi016 ceph-mon[29101]: from='client.24103 v1:172.21.15.183:0/1029239341' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi016=y;smithi183=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:13:41.020 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:40 smithi016 ceph-mon[29101]: Saving service mgr spec with placement smithi016=y;smithi183=x;count:2 2024-03-26T04:13:41.020 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:40 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:41.021 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:40 smithi016 ceph-mon[37215]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:41.021 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:40 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:41.021 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:40 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:41.021 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:40 smithi016 ceph-mon[37215]: Reconfiguring mon.b (monmap changed)... 2024-03-26T04:13:41.021 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:40 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:13:41.021 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:40 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:13:41.021 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:40 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:41.021 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:40 smithi016 ceph-mon[37215]: Reconfiguring daemon mon.b on smithi183 2024-03-26T04:13:41.021 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:40 smithi016 ceph-mon[37215]: from='client.24103 v1:172.21.15.183:0/1029239341' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi016=y;smithi183=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:13:41.021 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:40 smithi016 ceph-mon[37215]: Saving service mgr spec with placement smithi016=y;smithi183=x;count:2 2024-03-26T04:13:41.021 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:40 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:41.022 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-26T04:13:41.022 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:13:41.022 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-26T04:13:41.023 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:13:41.023 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:13:41.023 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-03-26 04:13:28.056457182 +0000 2024-03-26T04:13:41.023 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-03-26 04:03:46.534887913 +0000 2024-03-26T04:13:41.023 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-03-26 04:03:46.534887913 +0000 2024-03-26T04:13:41.023 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-03-26 04:03:46.534887913 +0000 2024-03-26T04:13:41.023 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-26T04:13:41.089 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-03-26T04:13:41.089 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-03-26T04:13:41.090 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.00010531 s, 4.9 MB/s 2024-03-26T04:13:41.090 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-26T04:13:41.146 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_4 2024-03-26T04:13:41.201 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-26T04:13:41.201 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:13:41.201 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-26T04:13:41.201 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:13:41.201 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:13:41.201 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-03-26 04:13:28.056457182 +0000 2024-03-26T04:13:41.201 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-03-26 04:03:46.868882149 +0000 2024-03-26T04:13:41.201 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-03-26 04:03:46.868882149 +0000 2024-03-26T04:13:41.201 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: 2024-03-26 04:03:46.868882149 +0000 2024-03-26T04:13:41.202 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-26T04:13:41.266 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-03-26T04:13:41.266 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-03-26T04:13:41.266 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000106312 s, 4.8 MB/s 2024-03-26T04:13:41.267 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-26T04:13:41.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:40 smithi183 ceph-mon[33745]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:41.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:40 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:41.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:40 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:41.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:40 smithi183 ceph-mon[33745]: Reconfiguring mon.b (monmap changed)... 2024-03-26T04:13:41.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:40 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-26T04:13:41.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:40 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-26T04:13:41.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:40 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:41.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:40 smithi183 ceph-mon[33745]: Reconfiguring daemon mon.b on smithi183 2024-03-26T04:13:41.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:40 smithi183 ceph-mon[33745]: from='client.24103 v1:172.21.15.183:0/1029239341' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi016=y;smithi183=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:13:41.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:40 smithi183 ceph-mon[33745]: Saving service mgr spec with placement smithi016=y;smithi183=x;count:2 2024-03-26T04:13:41.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:40 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:41.323 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-26T04:13:41.323 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/scratch_devs of=/dev/stdout 2024-03-26T04:13:41.339 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-26T04:13:41.339 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_1 2024-03-26T04:13:41.394 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-26T04:13:41.394 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:13:41.395 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-26T04:13:41.395 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:13:41.395 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:13:41.395 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-26 04:13:19.156730337 +0000 2024-03-26T04:13:41.395 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-26 04:03:45.857990390 +0000 2024-03-26T04:13:41.395 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-26 04:03:45.857990390 +0000 2024-03-26T04:13:41.395 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-26 04:03:45.857990390 +0000 2024-03-26T04:13:41.395 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-26T04:13:41.460 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-26T04:13:41.461 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-26T04:13:41.461 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000285921 s, 1.8 MB/s 2024-03-26T04:13:41.462 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-26T04:13:41.517 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_2 2024-03-26T04:13:41.573 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-26T04:13:41.573 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:13:41.573 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-26T04:13:41.573 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:13:41.573 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:13:41.573 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-26 04:13:19.157730310 +0000 2024-03-26T04:13:41.574 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-26 04:03:46.211980895 +0000 2024-03-26T04:13:41.574 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-26 04:03:46.211980895 +0000 2024-03-26T04:13:41.574 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-26 04:03:46.211980895 +0000 2024-03-26T04:13:41.574 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-26T04:13:41.642 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-26T04:13:41.642 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-26T04:13:41.642 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.00017639 s, 2.9 MB/s 2024-03-26T04:13:41.643 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-26T04:13:41.699 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_3 2024-03-26T04:13:41.753 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-26T04:13:41.753 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:13:41.753 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-26T04:13:41.754 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:13:41.754 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:13:41.754 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-26 04:13:19.157730310 +0000 2024-03-26T04:13:41.754 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-26 04:03:46.609970220 +0000 2024-03-26T04:13:41.754 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-26 04:03:46.609970220 +0000 2024-03-26T04:13:41.754 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-26 04:03:46.609970220 +0000 2024-03-26T04:13:41.754 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-26T04:13:41.819 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-26T04:13:41.819 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-26T04:13:41.820 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000197446 s, 2.6 MB/s 2024-03-26T04:13:41.821 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-26T04:13:41.878 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_4 2024-03-26T04:13:41.933 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-26T04:13:41.933 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-26T04:13:41.933 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-26T04:13:41.933 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-26T04:13:41.933 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-03-26T04:13:41.934 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-26 04:13:19.158730283 +0000 2024-03-26T04:13:41.934 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-26 04:03:46.969960563 +0000 2024-03-26T04:13:41.934 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-26 04:03:46.969960563 +0000 2024-03-26T04:13:41.934 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-26 04:03:46.969960563 +0000 2024-03-26T04:13:41.934 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-26T04:13:41.999 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-26T04:13:41.999 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-26T04:13:41.999 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000187022 s, 2.7 MB/s 2024-03-26T04:13:42.000 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-26T04:13:42.055 INFO:tasks.cephadm:Deploying osd.0 on smithi016 with /dev/vg_nvme/lv_4... 2024-03-26T04:13:42.056 DEBUG:teuthology.orchestra.run.smithi016:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-26T04:13:42.297 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:13:42.674 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:42 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:42.675 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:42 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:42.675 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:42 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:13:42.675 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:42 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:42.675 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:42 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:13:42.675 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:42 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:42.675 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:42 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:13:42.675 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:42 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T04:13:42.675 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:42 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:13:42.675 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:42 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:42.675 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:42 smithi016 ceph-mon[37215]: Deploying daemon mgr.x on smithi183 2024-03-26T04:13:42.675 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:42 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:42.676 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:42 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:42.676 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:42 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:13:42.676 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:42 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:42.676 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:42 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:13:42.676 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:42 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:42.676 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:42 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:13:42.676 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:42 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T04:13:42.676 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:42 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:13:42.676 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:42 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:42.676 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:42 smithi016 ceph-mon[29101]: Deploying daemon mgr.x on smithi183 2024-03-26T04:13:43.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:42 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:43.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:42 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:43.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:42 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:13:43.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:42 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:43.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:42 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:13:43.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:42 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:43.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:42 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:13:43.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:42 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-26T04:13:43.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:42 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:13:43.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:42 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:43.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:42 smithi183 ceph-mon[33745]: Deploying daemon mgr.x on smithi183 2024-03-26T04:13:43.671 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:13:43 smithi183 systemd[1]: Starting Ceph mgr.x for d7d59826-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:13:43.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:43 smithi016 ceph-mon[29101]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:43.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:43 smithi016 ceph-mon[37215]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:43.976 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:43 smithi183 ceph-mon[33745]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:43.976 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:13:43 smithi183 podman[35574]: 2024-03-26 04:13:43.774238596 +0000 UTC m=+0.109180798 container create be7c6ed09b1cbebdd09c182e6fb34d547ebd336c2c3a700bc85052c8dff0c7be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-26T04:13:43.976 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:13:43 smithi183 podman[35574]: 2024-03-26 04:13:43.682977687 +0000 UTC m=+0.017919888 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T04:13:43.976 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:13:43 smithi183 podman[35574]: 2024-03-26 04:13:43.976602063 +0000 UTC m=+0.311544270 container init be7c6ed09b1cbebdd09c182e6fb34d547ebd336c2c3a700bc85052c8dff0c7be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T04:13:44.058 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-26T04:13:44.058 INFO:teuthology.orchestra.run.smithi016.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-26T04:13:44.059 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10+0 records in 2024-03-26T04:13:44.059 INFO:teuthology.orchestra.run.smithi016.stderr:10+0 records out 2024-03-26T04:13:44.059 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00839421 s, 1.2 GB/s 2024-03-26T04:13:44.059 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping successful for: 2024-03-26T04:13:44.291 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:13:43 smithi183 podman[35574]: 2024-03-26 04:13:43.979178465 +0000 UTC m=+0.314120667 container start be7c6ed09b1cbebdd09c182e6fb34d547ebd336c2c3a700bc85052c8dff0c7be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240319) 2024-03-26T04:13:44.291 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:13:44 smithi183 ceph-mgr[35593]: -- 172.21.15.183:0/2996126367 <== mon.1 v1:172.21.15.183:6789/0 6 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (unknown 1147052007 0 0) 0x56215b5704e0 con 0x56215b546800 2024-03-26T04:13:44.291 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:13:44 smithi183 bash[35574]: be7c6ed09b1cbebdd09c182e6fb34d547ebd336c2c3a700bc85052c8dff0c7be 2024-03-26T04:13:44.291 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:13:44 smithi183 systemd[1]: Started Ceph mgr.x for d7d59826-eb26-11ee-95d0-87774f69a715. 2024-03-26T04:13:44.638 DEBUG:teuthology.orchestra.run.smithi016:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi016:vg_nvme/lv_4 2024-03-26T04:13:44.791 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:13:44 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:13:44.540+0000 7f758c2fe200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T04:13:44.791 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:13:44 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:13:44.637+0000 7f758c2fe200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T04:13:44.878 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:13:45.262 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:45 smithi016 ceph-mon[29101]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:45.262 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:45 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:45.262 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:45 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:45.262 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:45 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:45.263 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:45 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:45.263 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:45 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:13:45.263 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:45 smithi016 ceph-mon[37215]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:45.263 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:45 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:45.263 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:45 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:45.263 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:45 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:45.263 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:45 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:45.263 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:45 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:13:45.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:45 smithi183 ceph-mon[33745]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:45.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:45 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:45.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:45 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:45.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:45 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:45.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:45 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:45.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:45 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:13:45.656 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:13:45 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:13:45.312+0000 7f758c2fe200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T04:13:45.656 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:13:45 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:13:45.403+0000 7f758c2fe200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T04:13:45.656 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:13:45 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:13:45.558+0000 7f758c2fe200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T04:13:45.656 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:13:45 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:13:45.656+0000 7f758c2fe200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T04:13:46.000 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:13:45 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:13:45.738+0000 7f758c2fe200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T04:13:46.000 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:13:46 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:13:46.000+0000 7f758c2fe200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T04:13:46.289 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:13:46 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:13:46.107+0000 7f758c2fe200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T04:13:46.289 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:13:46 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:13:46.193+0000 7f758c2fe200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T04:13:46.289 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:13:46 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:13:46.288+0000 7f758c2fe200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T04:13:46.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:46.415+0000 7f2cecc30700 1 -- 172.21.15.16:0/2197438457 >> v1:172.21.15.16:6790/0 conn(0x7f2ce8106290 legacy=0x7f2ce8108680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:13:46.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:46.415+0000 7f2cecc30700 1 -- 172.21.15.16:0/2197438457 shutdown_connections 2024-03-26T04:13:46.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:46.415+0000 7f2cecc30700 1 -- 172.21.15.16:0/2197438457 >> 172.21.15.16:0/2197438457 conn(0x7f2ce80fd3f0 msgr2=0x7f2ce80ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:13:46.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:46.415+0000 7f2cecc30700 1 -- 172.21.15.16:0/2197438457 shutdown_connections 2024-03-26T04:13:46.416 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:46.416+0000 7f2cecc30700 1 -- 172.21.15.16:0/2197438457 wait complete. 2024-03-26T04:13:46.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:46.416+0000 7f2cecc30700 1 Processor -- start 2024-03-26T04:13:46.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:46.416+0000 7f2cecc30700 1 -- start start 2024-03-26T04:13:46.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:46.416+0000 7f2cecc30700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2ce81a4430 con 0x7f2ce8106290 2024-03-26T04:13:46.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:46.416+0000 7f2cecc30700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2ce81a5630 con 0x7f2ce8101e70 2024-03-26T04:13:46.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:46.416+0000 7f2cecc30700 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2ce81a6830 con 0x7f2ce8109f20 2024-03-26T04:13:46.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:46.417+0000 7f2ce5d9b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2ce8106290 0x7f2ce819f210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58326/0 (socket says 172.21.15.16:58326) 2024-03-26T04:13:46.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:46.417+0000 7f2ce5d9b700 1 -- 172.21.15.16:0/4281187164 learned_addr learned my addr 172.21.15.16:0/4281187164 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:13:46.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:46.417+0000 7f2cdf7fe700 1 -- 172.21.15.16:0/4281187164 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3123054728 0 0) 0x7f2ce81a6830 con 0x7f2ce8109f20 2024-03-26T04:13:46.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:46.417+0000 7f2cdf7fe700 1 -- 172.21.15.16:0/4281187164 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cd40035b0 con 0x7f2ce8109f20 2024-03-26T04:13:46.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:46.417+0000 7f2cdf7fe700 1 -- 172.21.15.16:0/4281187164 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1424660024 0 0) 0x7f2ce81a4430 con 0x7f2ce8106290 2024-03-26T04:13:46.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:46.417+0000 7f2cdf7fe700 1 -- 172.21.15.16:0/4281187164 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ce81a6830 con 0x7f2ce8106290 2024-03-26T04:13:46.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:46.417+0000 7f2cdf7fe700 1 -- 172.21.15.16:0/4281187164 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3716005310 0 0) 0x7f2ce81a5630 con 0x7f2ce8101e70 2024-03-26T04:13:46.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:46.417+0000 7f2cdf7fe700 1 -- 172.21.15.16:0/4281187164 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ce81a4430 con 0x7f2ce8101e70 2024-03-26T04:13:46.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:46.417+0000 7f2cdf7fe700 1 -- 172.21.15.16:0/4281187164 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 866050388 0 0) 0x7f2cd40035b0 con 0x7f2ce8109f20 2024-03-26T04:13:46.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:46.417+0000 7f2cdf7fe700 1 -- 172.21.15.16:0/4281187164 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ce81a5630 con 0x7f2ce8109f20 2024-03-26T04:13:46.419 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:46.417+0000 7f2cdf7fe700 1 -- 172.21.15.16:0/4281187164 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f2ce00018b0 con 0x7f2ce8109f20 2024-03-26T04:13:46.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:46.417+0000 7f2cdf7fe700 1 -- 172.21.15.16:0/4281187164 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 769991788 0 0) 0x7f2ce81a4430 con 0x7f2ce8101e70 2024-03-26T04:13:46.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:46.417+0000 7f2cdf7fe700 1 -- 172.21.15.16:0/4281187164 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2cd40035b0 con 0x7f2ce8101e70 2024-03-26T04:13:46.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:46.417+0000 7f2cdf7fe700 1 -- 172.21.15.16:0/4281187164 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2724031894 0 0) 0x7f2ce81a6830 con 0x7f2ce8106290 2024-03-26T04:13:46.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:46.417+0000 7f2cdf7fe700 1 -- 172.21.15.16:0/4281187164 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ce81a4430 con 0x7f2ce8106290 2024-03-26T04:13:46.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:46.417+0000 7f2cdf7fe700 1 -- 172.21.15.16:0/4281187164 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f2cd8001bb0 con 0x7f2ce8101e70 2024-03-26T04:13:46.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:46.417+0000 7f2cdf7fe700 1 -- 172.21.15.16:0/4281187164 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f2cd0001880 con 0x7f2ce8106290 2024-03-26T04:13:46.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:46.417+0000 7f2cdf7fe700 1 -- 172.21.15.16:0/4281187164 <== mon.2 v1:172.21.15.16:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3634181281 0 0) 0x7f2ce81a5630 con 0x7f2ce8109f20 2024-03-26T04:13:46.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:46.417+0000 7f2cdf7fe700 1 -- 172.21.15.16:0/4281187164 >> v1:172.21.15.183:6789/0 conn(0x7f2ce8101e70 legacy=0x7f2ce8075d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:13:46.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:46.417+0000 7f2cdf7fe700 1 -- 172.21.15.16:0/4281187164 >> v1:172.21.15.16:6789/0 conn(0x7f2ce8106290 legacy=0x7f2ce819f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:13:46.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:46.417+0000 7f2cdf7fe700 1 -- 172.21.15.16:0/4281187164 --> v1:172.21.15.16:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce81a7a30 con 0x7f2ce8109f20 2024-03-26T04:13:46.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:46.417+0000 7f2cecc30700 1 -- 172.21.15.16:0/4281187164 --> v1:172.21.15.16:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ce81a4620 con 0x7f2ce8109f20 2024-03-26T04:13:46.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:46.418+0000 7f2cecc30700 1 -- 172.21.15.16:0/4281187164 --> v1:172.21.15.16:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ce81a4b20 con 0x7f2ce8109f20 2024-03-26T04:13:46.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:46.418+0000 7f2cdf7fe700 1 -- 172.21.15.16:0/4281187164 <== mon.2 v1:172.21.15.16:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f2ce0003b30 con 0x7f2ce8109f20 2024-03-26T04:13:46.420 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:46.418+0000 7f2cdf7fe700 1 -- 172.21.15.16:0/4281187164 <== mon.2 v1:172.21.15.16:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f2ce0004a80 con 0x7f2ce8109f20 2024-03-26T04:13:46.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:46.418+0000 7f2cdf7fe700 1 -- 172.21.15.16:0/4281187164 <== mon.2 v1:172.21.15.16:6790/0 7 ==== mgrmap(e 13) v1 ==== 45949+0+0 (unknown 1805086598 0 0) 0x7f2ce000ff70 con 0x7f2ce8109f20 2024-03-26T04:13:46.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:46.418+0000 7f2cdf7fe700 1 -- 172.21.15.16:0/4281187164 <== mon.2 v1:172.21.15.16:6790/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 1069+0+0 (unknown 21079357 0 0) 0x7f2ce0045ed0 con 0x7f2ce8109f20 2024-03-26T04:13:46.421 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:46.419+0000 7f2cecc30700 1 -- 172.21.15.16:0/4281187164 --> v1:172.21.15.16:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cc8005150 con 0x7f2ce8109f20 2024-03-26T04:13:46.424 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:46.424+0000 7f2cdf7fe700 1 -- 172.21.15.16:0/4281187164 <== mon.2 v1:172.21.15.16:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f2ce00118e0 con 0x7f2ce8109f20 2024-03-26T04:13:46.541 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:13:46 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:13:46.465+0000 7f758c2fe200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T04:13:46.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:46 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:46.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:46 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:46.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:46 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:46.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:46 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:13:46.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:46 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:46.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:46 smithi183 ceph-mon[33745]: Reconfiguring mgr.y (unknown last config time)... 2024-03-26T04:13:46.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:46 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:13:46.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:46 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:13:46.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:46 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:46.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:46 smithi183 ceph-mon[33745]: Reconfiguring daemon mgr.y on smithi016 2024-03-26T04:13:46.641 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:46.641+0000 7f2cecc30700 1 -- 172.21.15.16:0/4281187164 --> v1:172.21.15.16:6800/4042489921 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f2cc8000bc0 con 0x7f2cd403a4b0 2024-03-26T04:13:46.739 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:46 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:46.739 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:46 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:46.739 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:46 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:46.740 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:46 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:13:46.740 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:46 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:46.740 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:46 smithi016 ceph-mon[29101]: Reconfiguring mgr.y (unknown last config time)... 2024-03-26T04:13:46.740 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:46 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:13:46.740 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:46 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:13:46.740 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:46 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:46.740 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:46 smithi016 ceph-mon[29101]: Reconfiguring daemon mgr.y on smithi016 2024-03-26T04:13:46.740 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:46 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:46.740 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:46 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:46.740 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:46 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:46.740 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:46 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:13:46.740 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:46 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:46.741 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:46 smithi016 ceph-mon[37215]: Reconfiguring mgr.y (unknown last config time)... 2024-03-26T04:13:46.741 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:46 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-26T04:13:46.741 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:46 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-26T04:13:46.741 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:46 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:46.741 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:46 smithi016 ceph-mon[37215]: Reconfiguring daemon mgr.y on smithi016 2024-03-26T04:13:47.600 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:47 smithi016 ceph-mon[29101]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:47.600 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:47 smithi016 ceph-mon[29101]: from='client.34115 v1:172.21.15.16:0/4281187164' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:13:47.600 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:47 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:13:47.600 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:47 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:13:47.600 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:47 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:47.600 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:47 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:47.600 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:47 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:47.600 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:47 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:13:47.600 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:47 smithi016 ceph-mon[37215]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:47.601 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:47 smithi016 ceph-mon[37215]: from='client.34115 v1:172.21.15.16:0/4281187164' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:13:47.601 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:47 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:13:47.601 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:47 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:13:47.601 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:47 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:47.601 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:47 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:47.601 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:47 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:47.601 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:47 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:13:47.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:47 smithi183 ceph-mon[33745]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:47.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:47 smithi183 ceph-mon[33745]: from='client.34115 v1:172.21.15.16:0/4281187164' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:13:47.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:47 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:13:47.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:47 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:13:47.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:47 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:47.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:47 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:47.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:47 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:47.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:47 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:13:47.792 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:13:47 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:13:47.420+0000 7f758c2fe200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T04:13:47.792 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:13:47 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:13:47.547+0000 7f758c2fe200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T04:13:48.170 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:13:47 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:13:47.886+0000 7f758c2fe200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T04:13:48.170 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:13:47 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:13:47.975+0000 7f758c2fe200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T04:13:48.170 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:13:48 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:13:48.169+0000 7f758c2fe200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:13:48.541 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:13:48 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:13:48.352+0000 7f758c2fe200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T04:13:48.903 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:13:48 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:13:48.633+0000 7f758c2fe200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:13:48.903 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:13:48 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:13:48.902+0000 7f758c2fe200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T04:13:49.291 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:13:49 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:13:49.135+0000 7f758c2fe200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T04:13:49.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:49 smithi016 ceph-mon[29101]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:49.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:49 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:49.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:49 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:13:49.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:49 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:49.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:49 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:49.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:49 smithi016 ceph-mon[37215]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:49.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:49 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:49.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:49 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:13:49.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:49 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:49.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:49 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:49.927 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:49 smithi183 ceph-mon[33745]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:49.927 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:49 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:49.927 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:49 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:13:49.927 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:49 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:49.927 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:49 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:13:50.294 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:13:49 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:13:49.926+0000 7f758c2fe200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T04:13:50.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:50 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/277688898' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e4e9d68-710d-4365-8810-e71718f37dee"}]: dispatch 2024-03-26T04:13:50.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:50 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/277688898' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5e4e9d68-710d-4365-8810-e71718f37dee"}]': finished 2024-03-26T04:13:50.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:50 smithi016 ceph-mon[29101]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T04:13:50.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:50 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:13:50.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:50 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/526052122' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:13:50.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:50 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/277688898' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e4e9d68-710d-4365-8810-e71718f37dee"}]: dispatch 2024-03-26T04:13:50.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:50 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/277688898' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5e4e9d68-710d-4365-8810-e71718f37dee"}]': finished 2024-03-26T04:13:50.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:50 smithi016 ceph-mon[37215]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T04:13:50.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:50 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:13:50.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:50 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/526052122' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:13:50.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:50 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/277688898' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5e4e9d68-710d-4365-8810-e71718f37dee"}]: dispatch 2024-03-26T04:13:50.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:50 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/277688898' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5e4e9d68-710d-4365-8810-e71718f37dee"}]': finished 2024-03-26T04:13:50.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:50 smithi183 ceph-mon[33745]: osdmap e5: 1 total, 0 up, 1 in 2024-03-26T04:13:50.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:50 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:13:50.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:50 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/526052122' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:13:50.792 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:13:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:13:50.491+0000 7f758c2fe200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T04:13:50.792 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:13:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:13:50.588+0000 7f758c2fe200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T04:13:51.291 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:13:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:13:50.799+0000 7f758c2fe200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T04:13:51.588 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:13:51.587+0000 7f2cdf7fe700 1 -- 172.21.15.16:0/4281187164 <== mon.2 v1:172.21.15.16:6790/0 10 ==== mgrmap(e 14) v1 ==== 91307+0+0 (unknown 816421452 0 0) 0x7f2ce0011560 con 0x7f2ce8109f20 2024-03-26T04:13:51.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:51 smithi016 ceph-mon[29101]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:51.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:51 smithi016 ceph-mon[29101]: Standby manager daemon x started 2024-03-26T04:13:51.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:51 smithi016 ceph-mon[29101]: from='mgr.? v1:172.21.15.183:0/2830418128' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T04:13:51.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:51 smithi016 ceph-mon[29101]: from='mgr.? v1:172.21.15.183:0/2830418128' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:13:51.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:51 smithi016 ceph-mon[29101]: from='mgr.? v1:172.21.15.183:0/2830418128' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T04:13:51.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:51 smithi016 ceph-mon[29101]: from='mgr.? v1:172.21.15.183:0/2830418128' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:13:51.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:51 smithi016 ceph-mon[37215]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:51.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:51 smithi016 ceph-mon[37215]: Standby manager daemon x started 2024-03-26T04:13:51.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:51 smithi016 ceph-mon[37215]: from='mgr.? v1:172.21.15.183:0/2830418128' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T04:13:51.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:51 smithi016 ceph-mon[37215]: from='mgr.? v1:172.21.15.183:0/2830418128' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:13:51.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:51 smithi016 ceph-mon[37215]: from='mgr.? v1:172.21.15.183:0/2830418128' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T04:13:51.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:51 smithi016 ceph-mon[37215]: from='mgr.? v1:172.21.15.183:0/2830418128' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:13:52.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:51 smithi183 ceph-mon[33745]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:52.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:51 smithi183 ceph-mon[33745]: Standby manager daemon x started 2024-03-26T04:13:52.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:51 smithi183 ceph-mon[33745]: from='mgr.? v1:172.21.15.183:0/2830418128' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T04:13:52.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:51 smithi183 ceph-mon[33745]: from='mgr.? v1:172.21.15.183:0/2830418128' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:13:52.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:51 smithi183 ceph-mon[33745]: from='mgr.? v1:172.21.15.183:0/2830418128' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T04:13:52.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:51 smithi183 ceph-mon[33745]: from='mgr.? v1:172.21.15.183:0/2830418128' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:13:52.844 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:52 smithi016 ceph-mon[29101]: mgrmap e14: y(active, since 67s), standbys: x 2024-03-26T04:13:52.845 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:52 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T04:13:52.845 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:52 smithi016 ceph-mon[37215]: mgrmap e14: y(active, since 67s), standbys: x 2024-03-26T04:13:52.845 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:52 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T04:13:53.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:52 smithi183 ceph-mon[33745]: mgrmap e14: y(active, since 67s), standbys: x 2024-03-26T04:13:53.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:52 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T04:13:53.876 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:53 smithi016 ceph-mon[29101]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:53.876 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:53 smithi016 ceph-mon[37215]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:54.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:53 smithi183 ceph-mon[33745]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:55.858 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:55 smithi183 ceph-mon[33745]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:55.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:55 smithi016 ceph-mon[29101]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:55.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:55 smithi016 ceph-mon[37215]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:56.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:56 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T04:13:56.856 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:56 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:56.856 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:56 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T04:13:56.857 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:56 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:57.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:56 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-26T04:13:57.039 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:56 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:13:57.853 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:57 smithi016 ceph-mon[29101]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:57.854 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:57 smithi016 ceph-mon[29101]: Deploying daemon osd.0 on smithi016 2024-03-26T04:13:57.854 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:57 smithi016 ceph-mon[37215]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:57.854 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:57 smithi016 ceph-mon[37215]: Deploying daemon osd.0 on smithi016 2024-03-26T04:13:58.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:57 smithi183 ceph-mon[33745]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:58.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:57 smithi183 ceph-mon[33745]: Deploying daemon osd.0 on smithi016 2024-03-26T04:13:59.701 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:13:59 smithi016 ceph-mon[29101]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:13:59.701 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:13:59 smithi016 ceph-mon[37215]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:14:00.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:13:59 smithi183 ceph-mon[33745]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:14:01.904 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:01 smithi016 ceph-mon[37215]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:14:01.904 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:01 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:01.904 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:01 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:01.904 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:01 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:14:01.904 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:01 smithi016 ceph-mon[29101]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:14:01.904 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:01 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:01.904 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:01 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:01.904 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:01 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:14:02.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:01 smithi183 ceph-mon[33745]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:14:02.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:01 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:02.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:01 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:02.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:01 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:14:03.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:03 smithi016 ceph-mon[29101]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:14:03.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:03 smithi016 ceph-mon[37215]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:14:04.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:03 smithi183 ceph-mon[33745]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:14:05.049 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:05.048+0000 7f2cdf7fe700 1 -- 172.21.15.16:0/4281187164 <== mgr.14152 v1:172.21.15.16:6800/4042489921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 275102686) 0x7f2cc8000bc0 con 0x7f2cd403a4b0 2024-03-26T04:14:05.050 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 0 on host 'smithi016' 2024-03-26T04:14:05.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:05.051+0000 7f2cecc30700 1 -- 172.21.15.16:0/4281187164 >> v1:172.21.15.16:6800/4042489921 conn(0x7f2cd403a4b0 legacy=0x7f2cd403c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:14:05.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:05.051+0000 7f2cecc30700 1 -- 172.21.15.16:0/4281187164 >> v1:172.21.15.16:6790/0 conn(0x7f2ce8109f20 legacy=0x7f2ce81a2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:14:05.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:05.051+0000 7f2cecc30700 1 -- 172.21.15.16:0/4281187164 shutdown_connections 2024-03-26T04:14:05.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:05.051+0000 7f2cecc30700 1 -- 172.21.15.16:0/4281187164 >> 172.21.15.16:0/4281187164 conn(0x7f2ce80fd3f0 msgr2=0x7f2ce810c3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:14:05.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:05.051+0000 7f2cecc30700 1 -- 172.21.15.16:0/4281187164 shutdown_connections 2024-03-26T04:14:05.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:05.051+0000 7f2cecc30700 1 -- 172.21.15.16:0/4281187164 wait complete. 2024-03-26T04:14:05.350 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:05 smithi016 ceph-mon[37215]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:14:05.350 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:05 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:05.351 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:05 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:05.351 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:05 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:14:05.351 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:05 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:14:05.351 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:05 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:05.351 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:05 smithi016 ceph-mon[37215]: from='osd.0 v1:172.21.15.16:6801/1132171550' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T04:14:05.351 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:05 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:05.351 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:05 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:05.351 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:05 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:14:05.351 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:05 smithi016 ceph-mon[29101]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:14:05.351 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:05 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:05.351 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:05 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:05.352 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:05 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:14:05.352 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:05 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:14:05.352 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:05 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:05.352 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:05 smithi016 ceph-mon[29101]: from='osd.0 v1:172.21.15.16:6801/1132171550' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T04:14:05.352 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:05 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:05.352 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:05 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:05.352 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:05 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:14:05.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:05 smithi183 ceph-mon[33745]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:14:05.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:05 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:05.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:05 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:05.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:05 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:14:05.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:05 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:14:05.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:05 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:05.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:05 smithi183 ceph-mon[33745]: from='osd.0 v1:172.21.15.16:6801/1132171550' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-26T04:14:05.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:05 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:05.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:05 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:05.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:05 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:14:05.620 DEBUG:teuthology.orchestra.run.smithi016:osd.0> sudo journalctl -f -n 0 -u ceph-d7d59826-eb26-11ee-95d0-87774f69a715@osd.0.service 2024-03-26T04:14:05.623 INFO:tasks.cephadm:Deploying osd.1 on smithi016 with /dev/vg_nvme/lv_3... 2024-03-26T04:14:05.623 DEBUG:teuthology.orchestra.run.smithi016:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-26T04:14:06.368 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:14:06.707 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:06 smithi016 ceph-mon[37215]: from='osd.0 v1:172.21.15.16:6801/1132171550' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T04:14:06.707 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:06 smithi016 ceph-mon[37215]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T04:14:06.707 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:06 smithi016 ceph-mon[37215]: from='osd.0 v1:172.21.15.16:6801/1132171550' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-03-26T04:14:06.707 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:06 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:14:06.707 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:06 smithi016 ceph-mon[29101]: from='osd.0 v1:172.21.15.16:6801/1132171550' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T04:14:06.708 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:06 smithi016 ceph-mon[29101]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T04:14:06.708 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:06 smithi016 ceph-mon[29101]: from='osd.0 v1:172.21.15.16:6801/1132171550' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-03-26T04:14:06.708 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:06 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:14:06.990 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 26 04:14:06 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-0[41205]: 2024-03-26T04:14:06.882+0000 7f08d0e8e700 -1 osd.0 0 waiting for initial osdmap 2024-03-26T04:14:06.991 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 26 04:14:06 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-0[41205]: 2024-03-26T04:14:06.884+0000 7f08c8c79700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T04:14:07.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:06 smithi183 ceph-mon[33745]: from='osd.0 v1:172.21.15.16:6801/1132171550' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-26T04:14:07.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:06 smithi183 ceph-mon[33745]: osdmap e6: 1 total, 0 up, 1 in 2024-03-26T04:14:07.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:06 smithi183 ceph-mon[33745]: from='osd.0 v1:172.21.15.16:6801/1132171550' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-03-26T04:14:07.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:06 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:14:07.800 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:07 smithi016 ceph-mon[29101]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:14:07.801 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:07 smithi016 ceph-mon[29101]: from='osd.0 v1:172.21.15.16:6801/1132171550' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-03-26T04:14:07.801 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:07 smithi016 ceph-mon[29101]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T04:14:07.801 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:07 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:14:07.801 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:07 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:14:07.801 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:07 smithi016 ceph-mon[29101]: from='osd.0 v1:172.21.15.16:6801/1132171550' entity='osd.0' 2024-03-26T04:14:07.801 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:07 smithi016 ceph-mon[37215]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:14:07.801 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:07 smithi016 ceph-mon[37215]: from='osd.0 v1:172.21.15.16:6801/1132171550' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-03-26T04:14:07.801 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:07 smithi016 ceph-mon[37215]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T04:14:07.801 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:07 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:14:07.801 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:07 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:14:07.801 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:07 smithi016 ceph-mon[37215]: from='osd.0 v1:172.21.15.16:6801/1132171550' entity='osd.0' 2024-03-26T04:14:08.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:07 smithi183 ceph-mon[33745]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:14:08.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:07 smithi183 ceph-mon[33745]: from='osd.0 v1:172.21.15.16:6801/1132171550' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-03-26T04:14:08.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:07 smithi183 ceph-mon[33745]: osdmap e7: 1 total, 0 up, 1 in 2024-03-26T04:14:08.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:07 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:14:08.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:07 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:14:08.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:07 smithi183 ceph-mon[33745]: from='osd.0 v1:172.21.15.16:6801/1132171550' entity='osd.0' 2024-03-26T04:14:08.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:08 smithi016 ceph-mon[29101]: purged_snaps scrub starts 2024-03-26T04:14:08.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:08 smithi016 ceph-mon[29101]: purged_snaps scrub ok 2024-03-26T04:14:08.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:08 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:14:08.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:08 smithi016 ceph-mon[29101]: osd.0 v1:172.21.15.16:6801/1132171550 boot 2024-03-26T04:14:08.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:08 smithi016 ceph-mon[29101]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T04:14:08.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:08 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:14:08.910 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:08 smithi016 ceph-mon[37215]: purged_snaps scrub starts 2024-03-26T04:14:08.910 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:08 smithi016 ceph-mon[37215]: purged_snaps scrub ok 2024-03-26T04:14:08.910 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:08 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:14:08.910 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:08 smithi016 ceph-mon[37215]: osd.0 v1:172.21.15.16:6801/1132171550 boot 2024-03-26T04:14:08.910 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:08 smithi016 ceph-mon[37215]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T04:14:08.910 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:08 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:14:09.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:08 smithi183 ceph-mon[33745]: purged_snaps scrub starts 2024-03-26T04:14:09.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:08 smithi183 ceph-mon[33745]: purged_snaps scrub ok 2024-03-26T04:14:09.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:08 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:14:09.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:08 smithi183 ceph-mon[33745]: osd.0 v1:172.21.15.16:6801/1132171550 boot 2024-03-26T04:14:09.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:08 smithi183 ceph-mon[33745]: osdmap e8: 1 total, 1 up, 1 in 2024-03-26T04:14:09.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:08 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:14:09.368 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-26T04:14:09.368 INFO:teuthology.orchestra.run.smithi016.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-26T04:14:09.368 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10+0 records in 2024-03-26T04:14:09.368 INFO:teuthology.orchestra.run.smithi016.stderr:10+0 records out 2024-03-26T04:14:09.368 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0107217 s, 978 MB/s 2024-03-26T04:14:09.368 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping successful for: 2024-03-26T04:14:09.911 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:09 smithi016 ceph-mon[29101]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:14:09.911 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:09 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:09.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:09 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:09.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:09 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:14:09.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:09 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:09.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:09 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:14:09.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:09 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:14:09.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:09 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:09.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:09 smithi016 ceph-mon[37215]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:14:09.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:09 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:09.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:09 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:09.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:09 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:14:09.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:09 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:09.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:09 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:14:09.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:09 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:14:09.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:09 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:10.015 DEBUG:teuthology.orchestra.run.smithi016:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi016:vg_nvme/lv_3 2024-03-26T04:14:10.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:09 smithi183 ceph-mon[33745]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-26T04:14:10.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:09 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:10.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:09 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:10.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:09 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:14:10.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:09 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:10.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:09 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:14:10.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:09 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:14:10.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:09 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:10.254 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:14:10.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:10 smithi016 ceph-mon[37215]: Detected new or changed devices on smithi016 2024-03-26T04:14:10.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:10 smithi016 ceph-mon[37215]: Adjusting osd_memory_target on smithi016 to 16069M 2024-03-26T04:14:10.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:10 smithi016 ceph-mon[37215]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T04:14:10.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:10 smithi016 ceph-mon[29101]: Detected new or changed devices on smithi016 2024-03-26T04:14:10.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:10 smithi016 ceph-mon[29101]: Adjusting osd_memory_target on smithi016 to 16069M 2024-03-26T04:14:10.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:10 smithi016 ceph-mon[29101]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T04:14:11.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:10 smithi183 ceph-mon[33745]: Detected new or changed devices on smithi016 2024-03-26T04:14:11.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:10 smithi183 ceph-mon[33745]: Adjusting osd_memory_target on smithi016 to 16069M 2024-03-26T04:14:11.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:10 smithi183 ceph-mon[33745]: osdmap e9: 1 total, 1 up, 1 in 2024-03-26T04:14:11.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:11.689+0000 7fecc7a45700 1 -- 172.21.15.16:0/3213962481 >> v1:172.21.15.16:6790/0 conn(0x7fecc0108ed0 legacy=0x7fecc010b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:14:11.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:11.689+0000 7fecc7a45700 1 -- 172.21.15.16:0/3213962481 shutdown_connections 2024-03-26T04:14:11.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:11.689+0000 7fecc7a45700 1 -- 172.21.15.16:0/3213962481 >> 172.21.15.16:0/3213962481 conn(0x7fecc00fbaf0 msgr2=0x7fecc00fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:14:11.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:11.690+0000 7fecc7a45700 1 -- 172.21.15.16:0/3213962481 shutdown_connections 2024-03-26T04:14:11.693 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:11.690+0000 7fecc7a45700 1 -- 172.21.15.16:0/3213962481 wait complete. 2024-03-26T04:14:11.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:11.690+0000 7fecc7a45700 1 Processor -- start 2024-03-26T04:14:11.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:11.690+0000 7fecc7a45700 1 -- start start 2024-03-26T04:14:11.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:11.690+0000 7fecc7a45700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fecc0198100 con 0x7fecc01052a0 2024-03-26T04:14:11.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:11.690+0000 7fecc7a45700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fecc01a3400 con 0x7fecc0102320 2024-03-26T04:14:11.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:11.690+0000 7fecc7a45700 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fecc01a45a0 con 0x7fecc0108ed0 2024-03-26T04:14:11.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:11.691+0000 7fecc4fe0700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fecc01052a0 0x7fecc019d5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55216/0 (socket says 172.21.15.16:55216) 2024-03-26T04:14:11.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:11.691+0000 7fecc5fe2700 1 --1- >> v1:172.21.15.16:6790/0 conn(0x7fecc0108ed0 0x7fecc01a0cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6790/0 says I am v1:172.21.15.16:60788/0 (socket says 172.21.15.16:60788) 2024-03-26T04:14:11.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:11.691+0000 7fecc4fe0700 1 -- 172.21.15.16:0/1004040344 learned_addr learned my addr 172.21.15.16:0/1004040344 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:14:11.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:11.691+0000 7fecb67fc700 1 -- 172.21.15.16:0/1004040344 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 451215511 0 0) 0x7fecc01a45a0 con 0x7fecc0108ed0 2024-03-26T04:14:11.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:11.691+0000 7fecb67fc700 1 -- 172.21.15.16:0/1004040344 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecac0035b0 con 0x7fecc0108ed0 2024-03-26T04:14:11.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:11.691+0000 7fecb67fc700 1 -- 172.21.15.16:0/1004040344 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1438776673 0 0) 0x7fecc0198100 con 0x7fecc01052a0 2024-03-26T04:14:11.694 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:11.691+0000 7fecb67fc700 1 -- 172.21.15.16:0/1004040344 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecc01a45a0 con 0x7fecc01052a0 2024-03-26T04:14:11.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:11.691+0000 7fecb67fc700 1 -- 172.21.15.16:0/1004040344 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2930641294 0 0) 0x7fecc01a3400 con 0x7fecc0102320 2024-03-26T04:14:11.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:11.691+0000 7fecb67fc700 1 -- 172.21.15.16:0/1004040344 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecc0198100 con 0x7fecc0102320 2024-03-26T04:14:11.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:11.691+0000 7fecb67fc700 1 -- 172.21.15.16:0/1004040344 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3316950696 0 0) 0x7fecc01a45a0 con 0x7fecc01052a0 2024-03-26T04:14:11.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:11.691+0000 7fecb67fc700 1 -- 172.21.15.16:0/1004040344 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fecc01a3400 con 0x7fecc01052a0 2024-03-26T04:14:11.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:11.691+0000 7fecb67fc700 1 -- 172.21.15.16:0/1004040344 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fecb0001970 con 0x7fecc01052a0 2024-03-26T04:14:11.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:11.691+0000 7fecb67fc700 1 -- 172.21.15.16:0/1004040344 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2869716069 0 0) 0x7fecac0035b0 con 0x7fecc0108ed0 2024-03-26T04:14:11.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:11.691+0000 7fecb67fc700 1 -- 172.21.15.16:0/1004040344 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fecc01a45a0 con 0x7fecc0108ed0 2024-03-26T04:14:11.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:11.691+0000 7fecb67fc700 1 -- 172.21.15.16:0/1004040344 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 436197759 0 0) 0x7fecc0198100 con 0x7fecc0102320 2024-03-26T04:14:11.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:11.691+0000 7fecb67fc700 1 -- 172.21.15.16:0/1004040344 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fecac0035b0 con 0x7fecc0102320 2024-03-26T04:14:11.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:11.691+0000 7fecb67fc700 1 -- 172.21.15.16:0/1004040344 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fecb80048f0 con 0x7fecc0108ed0 2024-03-26T04:14:11.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:11.691+0000 7fecb67fc700 1 -- 172.21.15.16:0/1004040344 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fecbc001830 con 0x7fecc0102320 2024-03-26T04:14:11.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:11.691+0000 7fecb67fc700 1 -- 172.21.15.16:0/1004040344 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 379340915 0 0) 0x7fecc01a3400 con 0x7fecc01052a0 2024-03-26T04:14:11.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:11.691+0000 7fecb67fc700 1 -- 172.21.15.16:0/1004040344 >> v1:172.21.15.16:6790/0 conn(0x7fecc0108ed0 legacy=0x7fecc01a0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:14:11.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:11.691+0000 7fecb67fc700 1 -- 172.21.15.16:0/1004040344 >> v1:172.21.15.183:6789/0 conn(0x7fecc0102320 legacy=0x7fecc0197590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:14:11.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:11.691+0000 7fecb67fc700 1 -- 172.21.15.16:0/1004040344 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc01a5740 con 0x7fecc01052a0 2024-03-26T04:14:11.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:11.691+0000 7fecc7a45700 1 -- 172.21.15.16:0/1004040344 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecc01a4790 con 0x7fecc01052a0 2024-03-26T04:14:11.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:11.691+0000 7fecc7a45700 1 -- 172.21.15.16:0/1004040344 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fecc01a4c90 con 0x7fecc01052a0 2024-03-26T04:14:11.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:11.692+0000 7fecb67fc700 1 -- 172.21.15.16:0/1004040344 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7fecb0001c90 con 0x7fecc01052a0 2024-03-26T04:14:11.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:11.692+0000 7fecb67fc700 1 -- 172.21.15.16:0/1004040344 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fecb0005c80 con 0x7fecc01052a0 2024-03-26T04:14:11.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:11.693+0000 7fecc7a45700 1 -- 172.21.15.16:0/1004040344 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feca4005150 con 0x7fecc01052a0 2024-03-26T04:14:11.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:11.693+0000 7fecb67fc700 1 -- 172.21.15.16:0/1004040344 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 14) v1 ==== 91307+0+0 (unknown 816421452 0 0) 0x7fecb0003930 con 0x7fecc01052a0 2024-03-26T04:14:11.696 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:11.694+0000 7fecb67fc700 1 -- 172.21.15.16:0/1004040344 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1541+0+0 (unknown 485016824 0 0) 0x7fecb0087430 con 0x7fecc01052a0 2024-03-26T04:14:11.698 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:11.698+0000 7fecb67fc700 1 -- 172.21.15.16:0/1004040344 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fecb0051fa0 con 0x7fecc01052a0 2024-03-26T04:14:11.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:11 smithi016 ceph-mon[37215]: pgmap v42: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:14:11.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:11 smithi016 ceph-mon[29101]: pgmap v42: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:14:11.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:11.912+0000 7fecc7a45700 1 -- 172.21.15.16:0/1004040344 --> v1:172.21.15.16:6800/4042489921 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7feca4000bc0 con 0x7fecac06e840 2024-03-26T04:14:12.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:11 smithi183 ceph-mon[33745]: pgmap v42: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:14:12.795 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:12 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:14:12.795 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:12 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:14:12.795 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:12 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:14:12.795 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:12 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:14:12.795 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:12 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:14:12.795 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:12 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:14:13.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:12 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:14:13.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:12 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:14:13.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:12 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:14:13.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:13 smithi016 ceph-mon[29101]: pgmap v43: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:14:13.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:13 smithi016 ceph-mon[29101]: from='client.14247 v1:172.21.15.16:0/1004040344' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:14:13.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:13 smithi016 ceph-mon[37215]: pgmap v43: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:14:13.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:13 smithi016 ceph-mon[37215]: from='client.14247 v1:172.21.15.16:0/1004040344' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:14:14.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:13 smithi183 ceph-mon[33745]: pgmap v43: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:14:14.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:13 smithi183 ceph-mon[33745]: from='client.14247 v1:172.21.15.16:0/1004040344' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:14:14.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:14 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/294147208' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "332e927f-79f4-4c33-b7f9-6d353d1721b0"}]: dispatch 2024-03-26T04:14:14.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:14 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/294147208' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "332e927f-79f4-4c33-b7f9-6d353d1721b0"}]': finished 2024-03-26T04:14:14.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:14 smithi016 ceph-mon[29101]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T04:14:14.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:14 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:14:14.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:14 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/294147208' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "332e927f-79f4-4c33-b7f9-6d353d1721b0"}]: dispatch 2024-03-26T04:14:14.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:14 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/294147208' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "332e927f-79f4-4c33-b7f9-6d353d1721b0"}]': finished 2024-03-26T04:14:14.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:14 smithi016 ceph-mon[37215]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T04:14:14.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:14 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:14:15.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:14 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/294147208' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "332e927f-79f4-4c33-b7f9-6d353d1721b0"}]: dispatch 2024-03-26T04:14:15.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:14 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/294147208' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "332e927f-79f4-4c33-b7f9-6d353d1721b0"}]': finished 2024-03-26T04:14:15.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:14 smithi183 ceph-mon[33745]: osdmap e10: 2 total, 1 up, 2 in 2024-03-26T04:14:15.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:14 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:14:15.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:15 smithi016 ceph-mon[29101]: pgmap v44: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:14:15.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:15 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/966611529' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:14:15.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:15 smithi016 ceph-mon[37215]: pgmap v44: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:14:15.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:15 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/966611529' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:14:16.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:15 smithi183 ceph-mon[33745]: pgmap v44: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:14:16.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:15 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/966611529' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:14:17.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:17 smithi016 ceph-mon[37215]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:14:17.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:17 smithi016 ceph-mon[29101]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:14:18.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:17 smithi183 ceph-mon[33745]: pgmap v46: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:14:19.766 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:19 smithi016 ceph-mon[29101]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:14:19.767 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:19 smithi016 ceph-mon[37215]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:14:20.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:19 smithi183 ceph-mon[33745]: pgmap v47: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:14:21.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:21 smithi016 ceph-mon[29101]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:14:21.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:21 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T04:14:21.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:21 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:14:21.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:21 smithi016 ceph-mon[29101]: Deploying daemon osd.1 on smithi016 2024-03-26T04:14:21.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:21 smithi016 ceph-mon[37215]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:14:21.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:21 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T04:14:21.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:21 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:14:21.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:21 smithi016 ceph-mon[37215]: Deploying daemon osd.1 on smithi016 2024-03-26T04:14:22.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:21 smithi183 ceph-mon[33745]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:14:22.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:21 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-26T04:14:22.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:21 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:14:22.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:21 smithi183 ceph-mon[33745]: Deploying daemon osd.1 on smithi016 2024-03-26T04:14:23.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:23 smithi016 ceph-mon[29101]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:14:23.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:23 smithi016 ceph-mon[37215]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:14:24.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:23 smithi183 ceph-mon[33745]: pgmap v49: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:14:25.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:25 smithi016 ceph-mon[29101]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:14:25.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:25 smithi016 ceph-mon[37215]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:14:26.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:25 smithi183 ceph-mon[33745]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:14:27.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:26 smithi016 ceph-mon[29101]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:14:27.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:26 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:27.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:26 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:27.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:26 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:14:27.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:26 smithi016 ceph-mon[37215]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:14:27.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:26 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:27.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:26 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:27.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:26 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:14:27.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:26 smithi183 ceph-mon[33745]: pgmap v51: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:14:27.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:26 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:27.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:26 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:27.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:26 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:14:29.616 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:29.615+0000 7fecb67fc700 1 -- 172.21.15.16:0/1004040344 <== mgr.14152 v1:172.21.15.16:6800/4042489921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 1282430314) 0x7feca4000bc0 con 0x7fecac06e840 2024-03-26T04:14:29.617 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 1 on host 'smithi016' 2024-03-26T04:14:29.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:29.619+0000 7fecc7a45700 1 -- 172.21.15.16:0/1004040344 >> v1:172.21.15.16:6800/4042489921 conn(0x7fecac06e840 legacy=0x7fecac070d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:14:29.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:29.619+0000 7fecc7a45700 1 -- 172.21.15.16:0/1004040344 >> v1:172.21.15.16:6789/0 conn(0x7fecc01052a0 legacy=0x7fecc019d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:14:29.619 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:29.619+0000 7fecc7a45700 1 -- 172.21.15.16:0/1004040344 shutdown_connections 2024-03-26T04:14:29.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:29.619+0000 7fecc7a45700 1 -- 172.21.15.16:0/1004040344 >> 172.21.15.16:0/1004040344 conn(0x7fecc00fbaf0 msgr2=0x7fecc0103290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:14:29.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:29.619+0000 7fecc7a45700 1 -- 172.21.15.16:0/1004040344 shutdown_connections 2024-03-26T04:14:29.620 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:29.619+0000 7fecc7a45700 1 -- 172.21.15.16:0/1004040344 wait complete. 2024-03-26T04:14:29.848 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:29 smithi016 ceph-mon[29101]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:14:29.848 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:29 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:29.848 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:29 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:29.848 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:29 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:14:29.849 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:29 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:14:29.849 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:29 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:29.849 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:29 smithi016 ceph-mon[29101]: from='osd.1 v1:172.21.15.16:6805/317987982' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T04:14:29.849 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:29 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:29.849 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:29 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:29.849 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:29 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:14:29.849 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:29 smithi016 ceph-mon[37215]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:14:29.849 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:29 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:29.849 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:29 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:29.849 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:29 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:14:29.849 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:29 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:14:29.849 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:29 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:29.850 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:29 smithi016 ceph-mon[37215]: from='osd.1 v1:172.21.15.16:6805/317987982' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T04:14:29.850 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:29 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:29.850 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:29 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:29.850 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:29 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:14:30.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:29 smithi183 ceph-mon[33745]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:14:30.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:29 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:30.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:29 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:30.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:29 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:14:30.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:29 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:14:30.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:29 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:30.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:29 smithi183 ceph-mon[33745]: from='osd.1 v1:172.21.15.16:6805/317987982' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-26T04:14:30.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:29 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:30.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:29 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:30.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:29 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:14:30.248 DEBUG:teuthology.orchestra.run.smithi016:osd.1> sudo journalctl -f -n 0 -u ceph-d7d59826-eb26-11ee-95d0-87774f69a715@osd.1.service 2024-03-26T04:14:30.251 INFO:tasks.cephadm:Deploying osd.2 on smithi016 with /dev/vg_nvme/lv_2... 2024-03-26T04:14:30.251 DEBUG:teuthology.orchestra.run.smithi016:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-26T04:14:30.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:30 smithi016 ceph-mon[29101]: from='osd.1 v1:172.21.15.16:6805/317987982' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T04:14:30.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:30 smithi016 ceph-mon[29101]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T04:14:30.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:30 smithi016 ceph-mon[29101]: from='osd.1 v1:172.21.15.16:6805/317987982' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-03-26T04:14:30.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:30 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:14:30.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:30 smithi016 ceph-mon[37215]: from='osd.1 v1:172.21.15.16:6805/317987982' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T04:14:30.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:30 smithi016 ceph-mon[37215]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T04:14:30.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:30 smithi016 ceph-mon[37215]: from='osd.1 v1:172.21.15.16:6805/317987982' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-03-26T04:14:30.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:30 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:14:30.913 INFO:journalctl@ceph.osd.1.smithi016.stdout:Mar 26 04:14:30 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-1[46918]: 2024-03-26T04:14:30.728+0000 7ff493736700 -1 osd.1 0 waiting for initial osdmap 2024-03-26T04:14:30.913 INFO:journalctl@ceph.osd.1.smithi016.stdout:Mar 26 04:14:30 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-1[46918]: 2024-03-26T04:14:30.731+0000 7ff48f529700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T04:14:31.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:30 smithi183 ceph-mon[33745]: from='osd.1 v1:172.21.15.16:6805/317987982' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-26T04:14:31.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:30 smithi183 ceph-mon[33745]: osdmap e11: 2 total, 1 up, 2 in 2024-03-26T04:14:31.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:30 smithi183 ceph-mon[33745]: from='osd.1 v1:172.21.15.16:6805/317987982' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-03-26T04:14:31.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:30 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:14:31.797 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:14:31.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:31 smithi016 ceph-mon[29101]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:14:31.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:31 smithi016 ceph-mon[29101]: from='osd.1 v1:172.21.15.16:6805/317987982' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-03-26T04:14:31.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:31 smithi016 ceph-mon[29101]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T04:14:31.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:31 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:14:31.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:31 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:14:31.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:31 smithi016 ceph-mon[37215]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:14:31.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:31 smithi016 ceph-mon[37215]: from='osd.1 v1:172.21.15.16:6805/317987982' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-03-26T04:14:31.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:31 smithi016 ceph-mon[37215]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T04:14:31.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:31 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:14:31.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:31 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:14:32.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:31 smithi183 ceph-mon[33745]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:14:32.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:31 smithi183 ceph-mon[33745]: from='osd.1 v1:172.21.15.16:6805/317987982' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-03-26T04:14:32.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:31 smithi183 ceph-mon[33745]: osdmap e12: 2 total, 1 up, 2 in 2024-03-26T04:14:32.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:31 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:14:32.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:31 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:14:32.992 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:32 smithi016 ceph-mon[37215]: purged_snaps scrub starts 2024-03-26T04:14:32.993 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:32 smithi016 ceph-mon[37215]: purged_snaps scrub ok 2024-03-26T04:14:32.993 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:32 smithi016 ceph-mon[37215]: osd.1 v1:172.21.15.16:6805/317987982 boot 2024-03-26T04:14:32.993 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:32 smithi016 ceph-mon[37215]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T04:14:32.993 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:32 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:14:32.993 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:32 smithi016 ceph-mon[29101]: purged_snaps scrub starts 2024-03-26T04:14:32.993 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:32 smithi016 ceph-mon[29101]: purged_snaps scrub ok 2024-03-26T04:14:32.993 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:32 smithi016 ceph-mon[29101]: osd.1 v1:172.21.15.16:6805/317987982 boot 2024-03-26T04:14:32.993 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:32 smithi016 ceph-mon[29101]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T04:14:32.993 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:32 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:14:33.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:32 smithi183 ceph-mon[33745]: purged_snaps scrub starts 2024-03-26T04:14:33.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:32 smithi183 ceph-mon[33745]: purged_snaps scrub ok 2024-03-26T04:14:33.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:32 smithi183 ceph-mon[33745]: osd.1 v1:172.21.15.16:6805/317987982 boot 2024-03-26T04:14:33.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:32 smithi183 ceph-mon[33745]: osdmap e13: 2 total, 2 up, 2 in 2024-03-26T04:14:33.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:32 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:14:33.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:33 smithi016 ceph-mon[29101]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:14:33.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:33 smithi016 ceph-mon[29101]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T04:14:33.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:33 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:33.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:33 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:33.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:33 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:14:33.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:33 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:14:33.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:33 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:33.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:33 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:14:33.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:33 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:14:33.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:33 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:33.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:33 smithi016 ceph-mon[37215]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:14:33.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:33 smithi016 ceph-mon[37215]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T04:14:33.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:33 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:33.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:33 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:33.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:33 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:14:33.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:33 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:14:33.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:33 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:33.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:33 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:14:33.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:33 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:14:33.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:33 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:34.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:33 smithi183 ceph-mon[33745]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-03-26T04:14:34.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:33 smithi183 ceph-mon[33745]: osdmap e14: 2 total, 2 up, 2 in 2024-03-26T04:14:34.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:33 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:34.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:33 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:34.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:33 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:14:34.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:33 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:14:34.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:33 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:34.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:33 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:14:34.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:33 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:14:34.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:33 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:34.212 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-26T04:14:34.212 INFO:teuthology.orchestra.run.smithi016.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-26T04:14:34.212 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10+0 records in 2024-03-26T04:14:34.212 INFO:teuthology.orchestra.run.smithi016.stderr:10+0 records out 2024-03-26T04:14:34.212 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00823563 s, 1.3 GB/s 2024-03-26T04:14:34.212 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping successful for: 2024-03-26T04:14:34.785 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:34 smithi016 ceph-mon[29101]: Detected new or changed devices on smithi016 2024-03-26T04:14:34.785 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:34 smithi016 ceph-mon[29101]: Adjusting osd_memory_target on smithi016 to 8034M 2024-03-26T04:14:34.785 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:34 smithi016 ceph-mon[37215]: Detected new or changed devices on smithi016 2024-03-26T04:14:34.785 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:34 smithi016 ceph-mon[37215]: Adjusting osd_memory_target on smithi016 to 8034M 2024-03-26T04:14:34.814 DEBUG:teuthology.orchestra.run.smithi016:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi016:vg_nvme/lv_2 2024-03-26T04:14:35.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:34 smithi183 ceph-mon[33745]: Detected new or changed devices on smithi016 2024-03-26T04:14:35.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:34 smithi183 ceph-mon[33745]: Adjusting osd_memory_target on smithi016 to 8034M 2024-03-26T04:14:35.056 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:14:36.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:35 smithi183 ceph-mon[33745]: pgmap v59: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:14:36.118 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:35 smithi016 ceph-mon[37215]: pgmap v59: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:14:36.118 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:35 smithi016 ceph-mon[29101]: pgmap v59: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:14:36.602 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:36.601+0000 7f3570155700 1 -- 172.21.15.16:0/2262253739 >> v1:172.21.15.16:6789/0 conn(0x7f3568101e90 legacy=0x7f35681022b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:14:36.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:36.602+0000 7f3570155700 1 -- 172.21.15.16:0/2262253739 shutdown_connections 2024-03-26T04:14:36.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:36.602+0000 7f3570155700 1 -- 172.21.15.16:0/2262253739 >> 172.21.15.16:0/2262253739 conn(0x7f35680fd430 msgr2=0x7f35680ff870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:14:36.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:36.602+0000 7f3570155700 1 -- 172.21.15.16:0/2262253739 shutdown_connections 2024-03-26T04:14:36.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:36.602+0000 7f3570155700 1 -- 172.21.15.16:0/2262253739 wait complete. 2024-03-26T04:14:36.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:36.602+0000 7f3570155700 1 Processor -- start 2024-03-26T04:14:36.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:36.602+0000 7f3570155700 1 -- start start 2024-03-26T04:14:36.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:36.602+0000 7f3570155700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f35681a42b0 con 0x7f35681062b0 2024-03-26T04:14:36.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:36.602+0000 7f3570155700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f35681a54b0 con 0x7f3568109f40 2024-03-26T04:14:36.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:36.602+0000 7f3570155700 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f35681a66b0 con 0x7f3568101e90 2024-03-26T04:14:36.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:36.602+0000 7f356def1700 1 --1- >> v1:172.21.15.16:6790/0 conn(0x7f3568101e90 0x7f3568075d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6790/0 says I am v1:172.21.15.16:49316/0 (socket says 172.21.15.16:49316) 2024-03-26T04:14:36.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:36.602+0000 7f356def1700 1 -- 172.21.15.16:0/3540714280 learned_addr learned my addr 172.21.15.16:0/3540714280 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:14:36.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:36.603+0000 7f355effd700 1 -- 172.21.15.16:0/3540714280 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1772456556 0 0) 0x7f35681a66b0 con 0x7f3568101e90 2024-03-26T04:14:36.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:36.603+0000 7f355effd700 1 -- 172.21.15.16:0/3540714280 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35540035b0 con 0x7f3568101e90 2024-03-26T04:14:36.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:36.603+0000 7f355effd700 1 -- 172.21.15.16:0/3540714280 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 872844465 0 0) 0x7f35681a42b0 con 0x7f35681062b0 2024-03-26T04:14:36.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:36.603+0000 7f355effd700 1 -- 172.21.15.16:0/3540714280 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35681a66b0 con 0x7f35681062b0 2024-03-26T04:14:36.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:36.603+0000 7f355effd700 1 -- 172.21.15.16:0/3540714280 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 782693188 0 0) 0x7f35681a54b0 con 0x7f3568109f40 2024-03-26T04:14:36.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:36.603+0000 7f355effd700 1 -- 172.21.15.16:0/3540714280 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35681a42b0 con 0x7f3568109f40 2024-03-26T04:14:36.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:36.603+0000 7f355effd700 1 -- 172.21.15.16:0/3540714280 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 413931225 0 0) 0x7f35540035b0 con 0x7f3568101e90 2024-03-26T04:14:36.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:36.603+0000 7f355effd700 1 -- 172.21.15.16:0/3540714280 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35681a54b0 con 0x7f3568101e90 2024-03-26T04:14:36.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:36.603+0000 7f355effd700 1 -- 172.21.15.16:0/3540714280 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f3564000fe0 con 0x7f3568101e90 2024-03-26T04:14:36.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:36.603+0000 7f355effd700 1 -- 172.21.15.16:0/3540714280 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4152919531 0 0) 0x7f35681a66b0 con 0x7f35681062b0 2024-03-26T04:14:36.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:36.603+0000 7f355effd700 1 -- 172.21.15.16:0/3540714280 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35540035b0 con 0x7f35681062b0 2024-03-26T04:14:36.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:36.603+0000 7f355effd700 1 -- 172.21.15.16:0/3540714280 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f3558001830 con 0x7f35681062b0 2024-03-26T04:14:36.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:36.603+0000 7f355effd700 1 -- 172.21.15.16:0/3540714280 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2614594496 0 0) 0x7f35681a42b0 con 0x7f3568109f40 2024-03-26T04:14:36.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:36.603+0000 7f355effd700 1 -- 172.21.15.16:0/3540714280 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35681a66b0 con 0x7f3568109f40 2024-03-26T04:14:36.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:36.603+0000 7f355effd700 1 -- 172.21.15.16:0/3540714280 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f3560001bb0 con 0x7f3568109f40 2024-03-26T04:14:36.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:36.603+0000 7f355effd700 1 -- 172.21.15.16:0/3540714280 <== mon.2 v1:172.21.15.16:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1838776698 0 0) 0x7f35681a54b0 con 0x7f3568101e90 2024-03-26T04:14:36.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:36.603+0000 7f355effd700 1 -- 172.21.15.16:0/3540714280 >> v1:172.21.15.183:6789/0 conn(0x7f3568109f40 legacy=0x7f35681a29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:14:36.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:36.603+0000 7f355effd700 1 -- 172.21.15.16:0/3540714280 >> v1:172.21.15.16:6789/0 conn(0x7f35681062b0 legacy=0x7f356819f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:14:36.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:36.603+0000 7f355effd700 1 -- 172.21.15.16:0/3540714280 --> v1:172.21.15.16:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35681a78b0 con 0x7f3568101e90 2024-03-26T04:14:36.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:36.603+0000 7f3570155700 1 -- 172.21.15.16:0/3540714280 --> v1:172.21.15.16:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35681a44a0 con 0x7f3568101e90 2024-03-26T04:14:36.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:36.603+0000 7f3570155700 1 -- 172.21.15.16:0/3540714280 --> v1:172.21.15.16:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f35681a49a0 con 0x7f3568101e90 2024-03-26T04:14:36.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:36.604+0000 7f355effd700 1 -- 172.21.15.16:0/3540714280 <== mon.2 v1:172.21.15.16:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f3564001210 con 0x7f3568101e90 2024-03-26T04:14:36.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:36.604+0000 7f355effd700 1 -- 172.21.15.16:0/3540714280 <== mon.2 v1:172.21.15.16:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f3564004f00 con 0x7f3568101e90 2024-03-26T04:14:36.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:36.604+0000 7f3570155700 1 -- 172.21.15.16:0/3540714280 --> v1:172.21.15.16:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f354c005150 con 0x7f3568101e90 2024-03-26T04:14:36.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:36.605+0000 7f355effd700 1 -- 172.21.15.16:0/3540714280 <== mon.2 v1:172.21.15.16:6790/0 7 ==== mgrmap(e 14) v1 ==== 91307+0+0 (unknown 816421452 0 0) 0x7f3564001800 con 0x7f3568101e90 2024-03-26T04:14:36.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:36.605+0000 7f355effd700 1 -- 172.21.15.16:0/3540714280 <== mon.2 v1:172.21.15.16:6790/0 8 ==== osd_map(14..14 src has 1..14) v4 ==== 1833+0+0 (unknown 3912524911 0 0) 0x7f35640878d0 con 0x7f3568101e90 2024-03-26T04:14:36.610 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:36.610+0000 7f355effd700 1 -- 172.21.15.16:0/3540714280 <== mon.2 v1:172.21.15.16:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f35640521e0 con 0x7f3568101e90 2024-03-26T04:14:36.828 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:36.827+0000 7f3570155700 1 -- 172.21.15.16:0/3540714280 --> v1:172.21.15.16:6800/4042489921 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f354c000bc0 con 0x7f355406e910 2024-03-26T04:14:38.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:37 smithi183 ceph-mon[33745]: pgmap v60: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:14:38.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:37 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:14:38.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:37 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:14:38.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:37 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:14:38.100 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:37 smithi016 ceph-mon[29101]: pgmap v60: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:14:38.100 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:37 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:14:38.100 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:37 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:14:38.100 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:37 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:14:38.101 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:37 smithi016 ceph-mon[37215]: pgmap v60: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:14:38.101 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:37 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:14:38.101 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:37 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:14:38.101 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:37 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:14:38.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:38 smithi016 ceph-mon[29101]: from='client.34169 v1:172.21.15.16:0/3540714280' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:14:38.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:38 smithi016 ceph-mon[37215]: from='client.34169 v1:172.21.15.16:0/3540714280' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:14:39.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:38 smithi183 ceph-mon[33745]: from='client.34169 v1:172.21.15.16:0/3540714280' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:14:40.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:39 smithi183 ceph-mon[33745]: pgmap v61: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:14:40.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:39 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3962703753' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b935affe-6e20-4226-87d9-12d4afe35679"}]: dispatch 2024-03-26T04:14:40.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:39 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3962703753' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b935affe-6e20-4226-87d9-12d4afe35679"}]': finished 2024-03-26T04:14:40.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:39 smithi183 ceph-mon[33745]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T04:14:40.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:39 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:14:40.125 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:39 smithi016 ceph-mon[29101]: pgmap v61: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:14:40.125 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:39 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3962703753' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b935affe-6e20-4226-87d9-12d4afe35679"}]: dispatch 2024-03-26T04:14:40.126 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:39 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3962703753' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b935affe-6e20-4226-87d9-12d4afe35679"}]': finished 2024-03-26T04:14:40.126 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:39 smithi016 ceph-mon[29101]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T04:14:40.126 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:39 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:14:40.126 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:39 smithi016 ceph-mon[37215]: pgmap v61: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:14:40.126 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:39 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3962703753' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b935affe-6e20-4226-87d9-12d4afe35679"}]: dispatch 2024-03-26T04:14:40.126 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:39 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3962703753' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b935affe-6e20-4226-87d9-12d4afe35679"}]': finished 2024-03-26T04:14:40.126 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:39 smithi016 ceph-mon[37215]: osdmap e15: 3 total, 2 up, 3 in 2024-03-26T04:14:40.126 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:39 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:14:41.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:40 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/147407728' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:14:41.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:40 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/147407728' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:14:41.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:40 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/147407728' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:14:42.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:41 smithi183 ceph-mon[33745]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:14:42.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:41 smithi016 ceph-mon[29101]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:14:42.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:41 smithi016 ceph-mon[37215]: pgmap v63: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:14:44.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:43 smithi183 ceph-mon[33745]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:14:44.141 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:43 smithi016 ceph-mon[37215]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:14:44.141 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:43 smithi016 ceph-mon[29101]: pgmap v64: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:14:45.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:45 smithi016 ceph-mon[37215]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:14:45.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:45 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T04:14:45.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:45 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:14:45.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:45 smithi016 ceph-mon[29101]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:14:45.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:45 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T04:14:45.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:45 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:14:46.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:45 smithi183 ceph-mon[33745]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:14:46.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:45 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-26T04:14:46.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:45 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:14:46.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:46 smithi016 ceph-mon[29101]: Deploying daemon osd.2 on smithi016 2024-03-26T04:14:46.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:46 smithi016 ceph-mon[37215]: Deploying daemon osd.2 on smithi016 2024-03-26T04:14:47.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:46 smithi183 ceph-mon[33745]: Deploying daemon osd.2 on smithi016 2024-03-26T04:14:48.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:47 smithi183 ceph-mon[33745]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:14:48.068 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:47 smithi016 ceph-mon[37215]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:14:48.069 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:47 smithi016 ceph-mon[29101]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:14:49.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:48 smithi183 ceph-mon[33745]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:14:49.080 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:48 smithi016 ceph-mon[29101]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:14:49.080 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:48 smithi016 ceph-mon[37215]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:14:51.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:50 smithi016 ceph-mon[29101]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:14:51.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:50 smithi016 ceph-mon[37215]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:14:51.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:50 smithi183 ceph-mon[33745]: pgmap v68: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:14:52.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:51 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:52.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:51 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:52.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:51 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:14:52.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:51 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:52.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:51 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:52.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:51 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:14:52.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:51 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:52.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:51 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:52.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:51 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:14:53.185 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:52 smithi016 ceph-mon[29101]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:14:53.185 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:52 smithi016 ceph-mon[37215]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:14:53.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:52 smithi183 ceph-mon[33745]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:14:54.228 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 2 on host 'smithi016' 2024-03-26T04:14:54.229 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:54.227+0000 7f355effd700 1 -- 172.21.15.16:0/3540714280 <== mgr.14152 v1:172.21.15.16:6800/4042489921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 2823710390) 0x7f354c000bc0 con 0x7f355406e910 2024-03-26T04:14:54.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:54.230+0000 7f3570155700 1 -- 172.21.15.16:0/3540714280 >> v1:172.21.15.16:6800/4042489921 conn(0x7f355406e910 legacy=0x7f3554070dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:14:54.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:54.230+0000 7f3570155700 1 -- 172.21.15.16:0/3540714280 >> v1:172.21.15.16:6790/0 conn(0x7f3568101e90 legacy=0x7f3568075d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:14:54.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:54.230+0000 7f3570155700 1 -- 172.21.15.16:0/3540714280 shutdown_connections 2024-03-26T04:14:54.230 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:54.230+0000 7f3570155700 1 -- 172.21.15.16:0/3540714280 >> 172.21.15.16:0/3540714280 conn(0x7f35680fd430 msgr2=0x7f356810c400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:14:54.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:54.230+0000 7f3570155700 1 -- 172.21.15.16:0/3540714280 shutdown_connections 2024-03-26T04:14:54.231 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:14:54.230+0000 7f3570155700 1 -- 172.21.15.16:0/3540714280 wait complete. 2024-03-26T04:14:54.784 DEBUG:teuthology.orchestra.run.smithi016:osd.2> sudo journalctl -f -n 0 -u ceph-d7d59826-eb26-11ee-95d0-87774f69a715@osd.2.service 2024-03-26T04:14:54.786 INFO:tasks.cephadm:Deploying osd.3 on smithi016 with /dev/vg_nvme/lv_1... 2024-03-26T04:14:54.787 DEBUG:teuthology.orchestra.run.smithi016:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-26T04:14:55.083 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:14:55.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:54 smithi016 ceph-mon[29101]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:14:55.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:54 smithi016 ceph-mon[29101]: from='osd.2 v1:172.21.15.16:6809/1107259766' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T04:14:55.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:54 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:55.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:54 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:55.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:54 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:55.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:54 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:55.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:54 smithi016 ceph-mon[37215]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:14:55.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:54 smithi016 ceph-mon[37215]: from='osd.2 v1:172.21.15.16:6809/1107259766' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T04:14:55.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:54 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:55.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:54 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:55.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:54 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:55.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:54 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:55.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:54 smithi183 ceph-mon[33745]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:14:55.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:54 smithi183 ceph-mon[33745]: from='osd.2 v1:172.21.15.16:6809/1107259766' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-26T04:14:55.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:54 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:55.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:54 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:55.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:54 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:55.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:54 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:14:56.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:55 smithi016 ceph-mon[37215]: from='osd.2 v1:172.21.15.16:6809/1107259766' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T04:14:56.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:55 smithi016 ceph-mon[37215]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T04:14:56.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:55 smithi016 ceph-mon[37215]: from='osd.2 v1:172.21.15.16:6809/1107259766' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-03-26T04:14:56.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:55 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:14:56.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:55 smithi016 ceph-mon[29101]: from='osd.2 v1:172.21.15.16:6809/1107259766' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T04:14:56.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:55 smithi016 ceph-mon[29101]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T04:14:56.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:55 smithi016 ceph-mon[29101]: from='osd.2 v1:172.21.15.16:6809/1107259766' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-03-26T04:14:56.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:55 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:14:56.163 INFO:journalctl@ceph.osd.2.smithi016.stdout:Mar 26 04:14:55 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-2[52956]: 2024-03-26T04:14:55.938+0000 7f0e81048700 -1 osd.2 0 waiting for initial osdmap 2024-03-26T04:14:56.163 INFO:journalctl@ceph.osd.2.smithi016.stdout:Mar 26 04:14:55 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-2[52956]: 2024-03-26T04:14:55.944+0000 7f0e7ce3b700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T04:14:56.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:55 smithi183 ceph-mon[33745]: from='osd.2 v1:172.21.15.16:6809/1107259766' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-26T04:14:56.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:55 smithi183 ceph-mon[33745]: osdmap e16: 3 total, 2 up, 3 in 2024-03-26T04:14:56.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:55 smithi183 ceph-mon[33745]: from='osd.2 v1:172.21.15.16:6809/1107259766' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-03-26T04:14:56.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:55 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:14:56.858 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-26T04:14:56.858 INFO:teuthology.orchestra.run.smithi016.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-26T04:14:56.858 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10+0 records in 2024-03-26T04:14:56.858 INFO:teuthology.orchestra.run.smithi016.stderr:10+0 records out 2024-03-26T04:14:56.858 INFO:teuthology.orchestra.run.smithi016.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00974929 s, 1.1 GB/s 2024-03-26T04:14:56.858 INFO:teuthology.orchestra.run.smithi016.stderr:--> Zapping successful for: 2024-03-26T04:14:57.111 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:56 smithi016 ceph-mon[37215]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:14:57.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:56 smithi016 ceph-mon[37215]: from='osd.2 v1:172.21.15.16:6809/1107259766' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-03-26T04:14:57.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:56 smithi016 ceph-mon[37215]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T04:14:57.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:56 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:14:57.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:56 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:14:57.112 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:56 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:14:57.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:56 smithi016 ceph-mon[29101]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:14:57.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:56 smithi016 ceph-mon[29101]: from='osd.2 v1:172.21.15.16:6809/1107259766' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-03-26T04:14:57.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:56 smithi016 ceph-mon[29101]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T04:14:57.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:56 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:14:57.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:56 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:14:57.112 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:56 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:14:57.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:56 smithi183 ceph-mon[33745]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:14:57.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:56 smithi183 ceph-mon[33745]: from='osd.2 v1:172.21.15.16:6809/1107259766' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-03-26T04:14:57.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:56 smithi183 ceph-mon[33745]: osdmap e17: 3 total, 2 up, 3 in 2024-03-26T04:14:57.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:56 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:14:57.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:56 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:14:57.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:56 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:14:57.686 DEBUG:teuthology.orchestra.run.smithi016:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi016:vg_nvme/lv_1 2024-03-26T04:14:58.403 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:58 smithi016 ceph-mon[37215]: purged_snaps scrub starts 2024-03-26T04:14:58.404 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:58 smithi016 ceph-mon[37215]: purged_snaps scrub ok 2024-03-26T04:14:58.404 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:58 smithi016 ceph-mon[37215]: osd.2 v1:172.21.15.16:6809/1107259766 boot 2024-03-26T04:14:58.404 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:58 smithi016 ceph-mon[37215]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T04:14:58.404 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:58 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:14:58.404 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:58 smithi016 ceph-mon[29101]: purged_snaps scrub starts 2024-03-26T04:14:58.404 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:58 smithi016 ceph-mon[29101]: purged_snaps scrub ok 2024-03-26T04:14:58.404 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:58 smithi016 ceph-mon[29101]: osd.2 v1:172.21.15.16:6809/1107259766 boot 2024-03-26T04:14:58.404 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:58 smithi016 ceph-mon[29101]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T04:14:58.404 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:58 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:14:58.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:58 smithi183 ceph-mon[33745]: purged_snaps scrub starts 2024-03-26T04:14:58.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:58 smithi183 ceph-mon[33745]: purged_snaps scrub ok 2024-03-26T04:14:58.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:58 smithi183 ceph-mon[33745]: osd.2 v1:172.21.15.16:6809/1107259766 boot 2024-03-26T04:14:58.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:58 smithi183 ceph-mon[33745]: osdmap e18: 3 total, 3 up, 3 in 2024-03-26T04:14:58.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:58 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:14:58.654 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:14:59.272 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:59 smithi016 ceph-mon[37215]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:14:59.272 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:59 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:14:59.272 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:59 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-26T04:14:59.272 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:59 smithi016 ceph-mon[37215]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T04:14:59.272 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:14:59 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:14:59.272 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:59 smithi016 ceph-mon[29101]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:14:59.272 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:59 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:14:59.272 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:59 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-26T04:14:59.272 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:59 smithi016 ceph-mon[29101]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T04:14:59.272 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:14:59 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:14:59.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:59 smithi183 ceph-mon[33745]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-26T04:14:59.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:59 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:14:59.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:59 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-26T04:14:59.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:59 smithi183 ceph-mon[33745]: osdmap e19: 3 total, 3 up, 3 in 2024-03-26T04:14:59.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:14:59 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:14:59.888 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 26 04:14:59 smithi016 sudo[56732]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T04:14:59.889 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 26 04:14:59 smithi016 sudo[56732]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:14:59.889 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 26 04:14:59 smithi016 sudo[56732]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:14:59.889 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 26 04:14:59 smithi016 sudo[56732]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:15:00.162 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 26 04:15:00 smithi016 sudo[56768]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T04:15:00.162 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 26 04:15:00 smithi016 sudo[56768]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:15:00.162 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 26 04:15:00 smithi016 sudo[56768]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:15:00.162 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 26 04:15:00 smithi016 sudo[56768]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:15:00.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:00 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T04:15:00.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:00 smithi016 ceph-mon[37215]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T04:15:00.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:00 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:00.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:00 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:00.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:00 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:15:00.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:00 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:15:00.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:00 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:15:00.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:00 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:00.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:00 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:15:00.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:00 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:15:00.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:00 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:00.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:00 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:00.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:00 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T04:15:00.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:00 smithi016 ceph-mon[29101]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T04:15:00.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:00 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:00.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:00 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:00.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:00 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:15:00.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:00 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:15:00.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:00 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:15:00.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:00 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:00.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:00 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:15:00.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:00 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:15:00.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:00 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:00.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:00 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:00.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:00 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-26T04:15:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:00 smithi183 ceph-mon[33745]: osdmap e20: 3 total, 3 up, 3 in 2024-03-26T04:15:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:00 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:00 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:00 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:15:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:00 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:15:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:00 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:15:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:00 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:00 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:15:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:00 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:15:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:00 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:00 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:00.600 INFO:journalctl@ceph.osd.1.smithi016.stdout:Mar 26 04:15:00 smithi016 sudo[56803]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T04:15:00.600 INFO:journalctl@ceph.osd.1.smithi016.stdout:Mar 26 04:15:00 smithi016 sudo[56803]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:15:00.600 INFO:journalctl@ceph.osd.1.smithi016.stdout:Mar 26 04:15:00 smithi016 sudo[56803]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:15:00.600 INFO:journalctl@ceph.osd.1.smithi016.stdout:Mar 26 04:15:00 smithi016 sudo[56803]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:15:00.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:00.834+0000 7ff004b4a700 1 -- 172.21.15.16:0/3493461851 >> v1:172.21.15.16:6790/0 conn(0x7ff000106300 legacy=0x7ff0001086f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:15:00.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:00.834+0000 7ff004b4a700 1 -- 172.21.15.16:0/3493461851 shutdown_connections 2024-03-26T04:15:00.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:00.834+0000 7ff004b4a700 1 -- 172.21.15.16:0/3493461851 >> 172.21.15.16:0/3493461851 conn(0x7ff0000fd480 msgr2=0x7ff0000ff8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:15:00.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:00.834+0000 7ff004b4a700 1 -- 172.21.15.16:0/3493461851 shutdown_connections 2024-03-26T04:15:00.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:00.834+0000 7ff004b4a700 1 -- 172.21.15.16:0/3493461851 wait complete. 2024-03-26T04:15:00.838 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:00.835+0000 7ff004b4a700 1 Processor -- start 2024-03-26T04:15:00.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:00.835+0000 7ff004b4a700 1 -- start start 2024-03-26T04:15:00.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:00.835+0000 7ff004b4a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff0001a43c0 con 0x7ff000101e50 2024-03-26T04:15:00.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:00.835+0000 7ff004b4a700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff0001a55c0 con 0x7ff000109f30 2024-03-26T04:15:00.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:00.835+0000 7ff004b4a700 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff0001a67c0 con 0x7ff000106300 2024-03-26T04:15:00.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:00.835+0000 7feffe59c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff000101e50 0x7ff000075d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35802/0 (socket says 172.21.15.16:35802) 2024-03-26T04:15:00.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:00.835+0000 7feffdd9b700 1 --1- >> v1:172.21.15.16:6790/0 conn(0x7ff000106300 0x7ff00019f1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6790/0 says I am v1:172.21.15.16:59982/0 (socket says 172.21.15.16:59982) 2024-03-26T04:15:00.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:00.836+0000 7feffe59c700 1 -- 172.21.15.16:0/4176552778 learned_addr learned my addr 172.21.15.16:0/4176552778 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:15:00.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:00.836+0000 7fefef7fe700 1 -- 172.21.15.16:0/4176552778 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1111733929 0 0) 0x7ff0001a67c0 con 0x7ff000106300 2024-03-26T04:15:00.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:00.836+0000 7fefef7fe700 1 -- 172.21.15.16:0/4176552778 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefe80035b0 con 0x7ff000106300 2024-03-26T04:15:00.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:00.836+0000 7fefef7fe700 1 -- 172.21.15.16:0/4176552778 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2936673872 0 0) 0x7ff0001a43c0 con 0x7ff000101e50 2024-03-26T04:15:00.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:00.836+0000 7fefef7fe700 1 -- 172.21.15.16:0/4176552778 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0001a67c0 con 0x7ff000101e50 2024-03-26T04:15:00.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:00.836+0000 7fefef7fe700 1 -- 172.21.15.16:0/4176552778 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3190182791 0 0) 0x7ff0001a55c0 con 0x7ff000109f30 2024-03-26T04:15:00.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:00.836+0000 7fefef7fe700 1 -- 172.21.15.16:0/4176552778 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0001a43c0 con 0x7ff000109f30 2024-03-26T04:15:00.839 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:00.836+0000 7fefef7fe700 1 -- 172.21.15.16:0/4176552778 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4087092030 0 0) 0x7fefe80035b0 con 0x7ff000106300 2024-03-26T04:15:00.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:00.836+0000 7fefef7fe700 1 -- 172.21.15.16:0/4176552778 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0001a55c0 con 0x7ff000106300 2024-03-26T04:15:00.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:00.836+0000 7fefef7fe700 1 -- 172.21.15.16:0/4176552778 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7feff4004540 con 0x7ff000106300 2024-03-26T04:15:00.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:00.836+0000 7fefef7fe700 1 -- 172.21.15.16:0/4176552778 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 448135157 0 0) 0x7ff0001a67c0 con 0x7ff000101e50 2024-03-26T04:15:00.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:00.836+0000 7fefef7fe700 1 -- 172.21.15.16:0/4176552778 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fefe80035b0 con 0x7ff000101e50 2024-03-26T04:15:00.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:00.836+0000 7fefef7fe700 1 -- 172.21.15.16:0/4176552778 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7feff00017b0 con 0x7ff000101e50 2024-03-26T04:15:00.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:00.836+0000 7fefef7fe700 1 -- 172.21.15.16:0/4176552778 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3120512507 0 0) 0x7ff0001a43c0 con 0x7ff000109f30 2024-03-26T04:15:00.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:00.836+0000 7fefef7fe700 1 -- 172.21.15.16:0/4176552778 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0001a67c0 con 0x7ff000109f30 2024-03-26T04:15:00.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:00.836+0000 7fefef7fe700 1 -- 172.21.15.16:0/4176552778 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7feff8001a80 con 0x7ff000109f30 2024-03-26T04:15:00.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:00.836+0000 7fefef7fe700 1 -- 172.21.15.16:0/4176552778 <== mon.2 v1:172.21.15.16:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1305179075 0 0) 0x7ff0001a55c0 con 0x7ff000106300 2024-03-26T04:15:00.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:00.836+0000 7fefef7fe700 1 -- 172.21.15.16:0/4176552778 >> v1:172.21.15.183:6789/0 conn(0x7ff000109f30 legacy=0x7ff0001a2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:15:00.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:00.836+0000 7fefef7fe700 1 -- 172.21.15.16:0/4176552778 >> v1:172.21.15.16:6789/0 conn(0x7ff000101e50 legacy=0x7ff000075d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:15:00.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:00.836+0000 7fefef7fe700 1 -- 172.21.15.16:0/4176552778 --> v1:172.21.15.16:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0001a79c0 con 0x7ff000106300 2024-03-26T04:15:00.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:00.836+0000 7ff004b4a700 1 -- 172.21.15.16:0/4176552778 --> v1:172.21.15.16:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0001a69b0 con 0x7ff000106300 2024-03-26T04:15:00.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:00.836+0000 7ff004b4a700 1 -- 172.21.15.16:0/4176552778 --> v1:172.21.15.16:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0001a6eb0 con 0x7ff000106300 2024-03-26T04:15:00.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:00.837+0000 7fefef7fe700 1 -- 172.21.15.16:0/4176552778 <== mon.2 v1:172.21.15.16:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7feff4001a70 con 0x7ff000106300 2024-03-26T04:15:00.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:00.837+0000 7fefef7fe700 1 -- 172.21.15.16:0/4176552778 <== mon.2 v1:172.21.15.16:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7feff4004e30 con 0x7ff000106300 2024-03-26T04:15:00.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:00.837+0000 7ff004b4a700 1 -- 172.21.15.16:0/4176552778 --> v1:172.21.15.16:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefe0005150 con 0x7ff000106300 2024-03-26T04:15:00.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:00.838+0000 7fefef7fe700 1 -- 172.21.15.16:0/4176552778 <== mon.2 v1:172.21.15.16:6790/0 7 ==== mgrmap(e 14) v1 ==== 91307+0+0 (unknown 816421452 0 0) 0x7feff4004f90 con 0x7ff000106300 2024-03-26T04:15:00.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:00.838+0000 7fefef7fe700 1 -- 172.21.15.16:0/4176552778 <== mon.2 v1:172.21.15.16:6790/0 8 ==== osd_map(21..21 src has 1..21) v4 ==== 2511+0+0 (unknown 4067065155 0 0) 0x7feff4087a30 con 0x7ff000106300 2024-03-26T04:15:00.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:00.843+0000 7fefef7fe700 1 -- 172.21.15.16:0/4176552778 <== mon.2 v1:172.21.15.16:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7feff4052100 con 0x7ff000106300 2024-03-26T04:15:00.859 INFO:journalctl@ceph.osd.1.smithi016.stdout:Mar 26 04:15:00 smithi016 sudo[56828]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T04:15:00.859 INFO:journalctl@ceph.osd.1.smithi016.stdout:Mar 26 04:15:00 smithi016 sudo[56828]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:15:00.859 INFO:journalctl@ceph.osd.1.smithi016.stdout:Mar 26 04:15:00 smithi016 sudo[56828]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:15:00.859 INFO:journalctl@ceph.osd.1.smithi016.stdout:Mar 26 04:15:00 smithi016 sudo[56828]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:15:01.060 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:01.058+0000 7ff004b4a700 1 -- 172.21.15.16:0/4176552778 --> v1:172.21.15.16:6800/4042489921 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fefe0000bc0 con 0x7fefe806e910 2024-03-26T04:15:01.115 INFO:journalctl@ceph.osd.2.smithi016.stdout:Mar 26 04:15:00 smithi016 sudo[56858]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T04:15:01.115 INFO:journalctl@ceph.osd.2.smithi016.stdout:Mar 26 04:15:00 smithi016 sudo[56858]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:15:01.115 INFO:journalctl@ceph.osd.2.smithi016.stdout:Mar 26 04:15:00 smithi016 sudo[56858]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:15:01.115 INFO:journalctl@ceph.osd.2.smithi016.stdout:Mar 26 04:15:00 smithi016 sudo[56858]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:15:01.129 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:01.128+0000 7fefef7fe700 1 -- 172.21.15.16:0/4176552778 <== mon.2 v1:172.21.15.16:6790/0 10 ==== mgrmap(e 15) v1 ==== 91363+0+0 (unknown 160314185 0 0) 0x7feff4051bb0 con 0x7ff000106300 2024-03-26T04:15:01.388 INFO:journalctl@ceph.osd.2.smithi016.stdout:Mar 26 04:15:01 smithi016 sudo[56926]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T04:15:01.388 INFO:journalctl@ceph.osd.2.smithi016.stdout:Mar 26 04:15:01 smithi016 sudo[56926]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:15:01.388 INFO:journalctl@ceph.osd.2.smithi016.stdout:Mar 26 04:15:01 smithi016 sudo[56926]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:15:01.388 INFO:journalctl@ceph.osd.2.smithi016.stdout:Mar 26 04:15:01 smithi016 sudo[56926]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:15:01.388 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:01 smithi016 ceph-mon[37215]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:15:01.388 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:01 smithi016 ceph-mon[37215]: Detected new or changed devices on smithi016 2024-03-26T04:15:01.389 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:01 smithi016 ceph-mon[37215]: Adjusting osd_memory_target on smithi016 to 5356M 2024-03-26T04:15:01.389 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:01 smithi016 ceph-mon[37215]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T04:15:01.389 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:01 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:15:01.389 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:01 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:15:01.389 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:01 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:15:01.389 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:01 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:01.389 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:01 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:15:01.389 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:01 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:15:01.389 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:01 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:15:01.389 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:01 smithi016 ceph-mon[29101]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:15:01.389 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:01 smithi016 ceph-mon[29101]: Detected new or changed devices on smithi016 2024-03-26T04:15:01.389 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:01 smithi016 ceph-mon[29101]: Adjusting osd_memory_target on smithi016 to 5356M 2024-03-26T04:15:01.390 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:01 smithi016 ceph-mon[29101]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T04:15:01.390 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:01 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:15:01.390 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:01 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:15:01.390 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:01 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:15:01.390 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:01 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:01.390 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:01 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:15:01.390 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:01 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:15:01.390 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:01 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:15:01.390 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:01 smithi016 sudo[56931]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T04:15:01.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:01 smithi183 ceph-mon[33745]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:15:01.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:01 smithi183 ceph-mon[33745]: Detected new or changed devices on smithi016 2024-03-26T04:15:01.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:01 smithi183 ceph-mon[33745]: Adjusting osd_memory_target on smithi016 to 5356M 2024-03-26T04:15:01.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:01 smithi183 ceph-mon[33745]: osdmap e21: 3 total, 3 up, 3 in 2024-03-26T04:15:01.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:01 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:15:01.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:01 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:15:01.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:01 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:15:01.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:01 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:01.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:01 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:15:01.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:01 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:15:01.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:01 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:15:01.657 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:01 smithi016 sudo[56931]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:15:01.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:01 smithi016 sudo[56931]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:15:01.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:01 smithi016 sudo[56931]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:15:01.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:01 smithi016 sudo[56988]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T04:15:01.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:01 smithi016 sudo[56988]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:15:01.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:01 smithi016 sudo[56988]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:15:01.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:01 smithi016 sudo[56988]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:15:02.153 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:01 smithi183 sudo[36309]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T04:15:02.153 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:01 smithi183 sudo[36309]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:15:02.153 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:01 smithi183 sudo[36309]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:15:02.153 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:01 smithi183 sudo[36309]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:15:02.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:02 smithi016 ceph-mon[37215]: from='client.34202 v1:172.21.15.16:0/4176552778' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:15:02.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:02 smithi016 ceph-mon[37215]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-26T04:15:02.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:02 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:15:02.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:02 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:15:02.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:02 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:15:02.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:02 smithi016 ceph-mon[37215]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:15:02.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:02 smithi016 ceph-mon[37215]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:15:02.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:02 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:15:02.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:02 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:15:02.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:02 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:15:02.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:02 smithi016 ceph-mon[29101]: from='client.34202 v1:172.21.15.16:0/4176552778' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:15:02.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:02 smithi016 ceph-mon[29101]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-26T04:15:02.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:02 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:15:02.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:02 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:15:02.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:02 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:15:02.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:02 smithi016 ceph-mon[29101]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:15:02.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:02 smithi016 ceph-mon[29101]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:15:02.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:02 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:15:02.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:02 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:15:02.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:02 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:15:02.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:02 smithi183 sudo[36312]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T04:15:02.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:02 smithi183 sudo[36312]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:15:02.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:02 smithi183 sudo[36312]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:15:02.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:02 smithi183 sudo[36312]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:15:02.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:02 smithi183 ceph-mon[33745]: from='client.34202 v1:172.21.15.16:0/4176552778' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi016:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:15:02.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:02 smithi183 ceph-mon[33745]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-26T04:15:02.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:02 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:15:02.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:02 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:15:02.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:02 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:15:02.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:02 smithi183 ceph-mon[33745]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:15:02.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:02 smithi183 ceph-mon[33745]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:15:02.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:02 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:15:02.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:02 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:15:02.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:02 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:15:02.692 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:02 smithi016 sudo[57012]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-26T04:15:02.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:02 smithi016 sudo[57012]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:15:02.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:02 smithi016 sudo[57012]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:15:02.693 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:02 smithi016 sudo[57012]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:15:02.956 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:02 smithi016 sudo[57036]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-26T04:15:02.956 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:02 smithi016 sudo[57036]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-26T04:15:02.956 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:02 smithi016 sudo[57036]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-26T04:15:02.956 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:02 smithi016 sudo[57036]: pam_unix(sudo:session): session closed for user root 2024-03-26T04:15:03.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:03 smithi016 ceph-mon[29101]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:15:03.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:03 smithi016 ceph-mon[29101]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:15:03.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:03 smithi016 ceph-mon[29101]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:15:03.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:03 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:03.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:03 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:15:03.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:03 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:15:03.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:03 smithi016 ceph-mon[29101]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:15:03.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:03 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:15:03.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:03 smithi016 ceph-mon[29101]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:15:03.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:03 smithi016 ceph-mon[37215]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:15:03.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:03 smithi016 ceph-mon[37215]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:15:03.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:03 smithi016 ceph-mon[37215]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:15:03.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:03 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:03.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:03 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:15:03.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:03 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:15:03.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:03 smithi016 ceph-mon[37215]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:15:03.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:03 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:15:03.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:03 smithi016 ceph-mon[37215]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:15:03.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:03 smithi183 ceph-mon[33745]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:15:03.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:03 smithi183 ceph-mon[33745]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:15:03.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:03 smithi183 ceph-mon[33745]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:15:03.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:03 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:03.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:03 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:15:03.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:03 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:15:03.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:03 smithi183 ceph-mon[33745]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-26T04:15:03.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:03 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:15:03.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:03 smithi183 ceph-mon[33745]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-26T04:15:04.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:04 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3042305266' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c84a303-c00a-4a27-82c1-5e9ec1fb8d6c"}]: dispatch 2024-03-26T04:15:04.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:04 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3042305266' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6c84a303-c00a-4a27-82c1-5e9ec1fb8d6c"}]': finished 2024-03-26T04:15:04.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:04 smithi183 ceph-mon[33745]: osdmap e22: 4 total, 3 up, 4 in 2024-03-26T04:15:04.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:04 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:15:04.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:04 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3042305266' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c84a303-c00a-4a27-82c1-5e9ec1fb8d6c"}]: dispatch 2024-03-26T04:15:04.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:04 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3042305266' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6c84a303-c00a-4a27-82c1-5e9ec1fb8d6c"}]': finished 2024-03-26T04:15:04.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:04 smithi016 ceph-mon[29101]: osdmap e22: 4 total, 3 up, 4 in 2024-03-26T04:15:04.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:04 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:15:04.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:04 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3042305266' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c84a303-c00a-4a27-82c1-5e9ec1fb8d6c"}]: dispatch 2024-03-26T04:15:04.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:04 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3042305266' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6c84a303-c00a-4a27-82c1-5e9ec1fb8d6c"}]': finished 2024-03-26T04:15:04.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:04 smithi016 ceph-mon[37215]: osdmap e22: 4 total, 3 up, 4 in 2024-03-26T04:15:04.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:04 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:15:05.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:05 smithi183 ceph-mon[33745]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:15:05.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:05 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3418150832' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:15:05.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:05 smithi016 ceph-mon[37215]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:15:05.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:05 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3418150832' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:15:05.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:05 smithi016 ceph-mon[29101]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:15:05.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:05 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3418150832' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:15:07.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:07 smithi016 ceph-mon[29101]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:15:07.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:07 smithi016 ceph-mon[37215]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:15:07.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:07 smithi183 ceph-mon[33745]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:15:09.430 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:09 smithi016 ceph-mon[37215]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:15:09.431 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:09 smithi016 ceph-mon[29101]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:15:09.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:09 smithi183 ceph-mon[33745]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:15:10.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:10 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T04:15:10.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:10 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:15:10.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:10 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T04:15:10.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:10 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:15:10.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:10 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-26T04:15:10.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:10 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:15:11.501 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:11 smithi016 ceph-mon[37215]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:15:11.501 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:11 smithi016 ceph-mon[37215]: Deploying daemon osd.3 on smithi016 2024-03-26T04:15:11.502 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:11 smithi016 ceph-mon[29101]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:15:11.502 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:11 smithi016 ceph-mon[29101]: Deploying daemon osd.3 on smithi016 2024-03-26T04:15:11.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:11 smithi183 ceph-mon[33745]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:15:11.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:11 smithi183 ceph-mon[33745]: Deploying daemon osd.3 on smithi016 2024-03-26T04:15:13.351 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:13 smithi016 ceph-mon[37215]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:15:13.352 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:13 smithi016 ceph-mon[29101]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:15:13.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:13 smithi183 ceph-mon[33745]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:15:15.230 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:15 smithi016 ceph-mon[29101]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:15:15.230 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:15 smithi016 ceph-mon[37215]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:15:15.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:15 smithi183 ceph-mon[33745]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:15:16.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:16 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:16.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:16 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:16.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:16 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:15:16.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:16 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:16.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:16 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:16.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:16 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:15:16.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:16 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:16.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:16 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:16.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:16 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:15:17.495 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:17 smithi016 ceph-mon[37215]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:15:17.495 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:17 smithi016 ceph-mon[29101]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:15:17.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:17 smithi183 ceph-mon[33745]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:15:18.962 INFO:teuthology.orchestra.run.smithi016.stdout:Created osd(s) 3 on host 'smithi016' 2024-03-26T04:15:18.962 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:18.961+0000 7fefef7fe700 1 -- 172.21.15.16:0/4176552778 <== mgr.14152 v1:172.21.15.16:6800/4042489921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 4099639298) 0x7fefe0000bc0 con 0x7fefe806e910 2024-03-26T04:15:18.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:18.963+0000 7ff004b4a700 1 -- 172.21.15.16:0/4176552778 >> v1:172.21.15.16:6800/4042489921 conn(0x7fefe806e910 legacy=0x7fefe8070dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:15:18.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:18.963+0000 7ff004b4a700 1 -- 172.21.15.16:0/4176552778 >> v1:172.21.15.16:6790/0 conn(0x7ff000106300 legacy=0x7ff00019f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:15:18.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:18.963+0000 7ff004b4a700 1 -- 172.21.15.16:0/4176552778 shutdown_connections 2024-03-26T04:15:18.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:18.964+0000 7ff004b4a700 1 -- 172.21.15.16:0/4176552778 >> 172.21.15.16:0/4176552778 conn(0x7ff0000fd480 msgr2=0x7ff0000ff870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:15:18.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:18.964+0000 7ff004b4a700 1 -- 172.21.15.16:0/4176552778 shutdown_connections 2024-03-26T04:15:18.964 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:15:18.964+0000 7ff004b4a700 1 -- 172.21.15.16:0/4176552778 wait complete. 2024-03-26T04:15:19.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:19 smithi183 ceph-mon[33745]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:15:19.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:19 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:19.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:19 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:19.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:19 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:15:19.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:19 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:15:19.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:19 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:19.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:19 smithi183 ceph-mon[33745]: from='osd.3 v1:172.21.15.16:6813/1463789444' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T04:15:19.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:19 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:19.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:19 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:19.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:19 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:15:19.597 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:19 smithi016 ceph-mon[37215]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:15:19.597 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:19 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:19.597 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:19 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:19.597 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:19 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:15:19.598 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:19 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:15:19.598 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:19 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:19.598 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:19 smithi016 ceph-mon[37215]: from='osd.3 v1:172.21.15.16:6813/1463789444' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T04:15:19.598 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:19 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:19.598 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:19 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:19.598 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:19 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:15:19.598 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:19 smithi016 ceph-mon[29101]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:15:19.598 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:19 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:19.598 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:19 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:19.599 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:19 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:15:19.599 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:19 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:15:19.599 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:19 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:19.599 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:19 smithi016 ceph-mon[29101]: from='osd.3 v1:172.21.15.16:6813/1463789444' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-26T04:15:19.599 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:19 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:19.599 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:19 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:19.599 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:19 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:15:19.624 DEBUG:teuthology.orchestra.run.smithi016:osd.3> sudo journalctl -f -n 0 -u ceph-d7d59826-eb26-11ee-95d0-87774f69a715@osd.3.service 2024-03-26T04:15:19.626 INFO:tasks.cephadm:Deploying osd.4 on smithi183 with /dev/vg_nvme/lv_4... 2024-03-26T04:15:19.626 DEBUG:teuthology.orchestra.run.smithi183:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-26T04:15:19.873 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:15:20.637 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:20 smithi016 ceph-mon[37215]: from='osd.3 v1:172.21.15.16:6813/1463789444' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T04:15:20.637 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:20 smithi016 ceph-mon[37215]: osdmap e23: 4 total, 3 up, 4 in 2024-03-26T04:15:20.637 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:20 smithi016 ceph-mon[37215]: from='osd.3 v1:172.21.15.16:6813/1463789444' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-03-26T04:15:20.637 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:20 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:15:20.637 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:20 smithi016 ceph-mon[29101]: from='osd.3 v1:172.21.15.16:6813/1463789444' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T04:15:20.637 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:20 smithi016 ceph-mon[29101]: osdmap e23: 4 total, 3 up, 4 in 2024-03-26T04:15:20.637 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:20 smithi016 ceph-mon[29101]: from='osd.3 v1:172.21.15.16:6813/1463789444' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-03-26T04:15:20.637 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:20 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:15:20.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:20 smithi183 ceph-mon[33745]: from='osd.3 v1:172.21.15.16:6813/1463789444' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-26T04:15:20.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:20 smithi183 ceph-mon[33745]: osdmap e23: 4 total, 3 up, 4 in 2024-03-26T04:15:20.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:20 smithi183 ceph-mon[33745]: from='osd.3 v1:172.21.15.16:6813/1463789444' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]: dispatch 2024-03-26T04:15:20.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:20 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:15:20.911 INFO:journalctl@ceph.osd.3.smithi016.stdout:Mar 26 04:15:20 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-3[58771]: 2024-03-26T04:15:20.653+0000 7f839fa60700 -1 osd.3 0 waiting for initial osdmap 2024-03-26T04:15:20.912 INFO:journalctl@ceph.osd.3.smithi016.stdout:Mar 26 04:15:20 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-3[58771]: 2024-03-26T04:15:20.658+0000 7f839784b700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T04:15:21.655 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-26T04:15:21.655 INFO:teuthology.orchestra.run.smithi183.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-26T04:15:21.655 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10+0 records in 2024-03-26T04:15:21.655 INFO:teuthology.orchestra.run.smithi183.stderr:10+0 records out 2024-03-26T04:15:21.655 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110813 s, 946 MB/s 2024-03-26T04:15:21.655 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping successful for: 2024-03-26T04:15:21.656 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:21 smithi183 ceph-mon[33745]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:15:21.656 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:21 smithi183 ceph-mon[33745]: from='osd.3 v1:172.21.15.16:6813/1463789444' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-03-26T04:15:21.656 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:21 smithi183 ceph-mon[33745]: osdmap e24: 4 total, 3 up, 4 in 2024-03-26T04:15:21.656 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:21 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:15:21.656 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:21 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:15:21.656 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:21 smithi183 ceph-mon[33745]: from='osd.3 v1:172.21.15.16:6813/1463789444' entity='osd.3' 2024-03-26T04:15:21.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:21 smithi016 ceph-mon[29101]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:15:21.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:21 smithi016 ceph-mon[29101]: from='osd.3 v1:172.21.15.16:6813/1463789444' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-03-26T04:15:21.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:21 smithi016 ceph-mon[29101]: osdmap e24: 4 total, 3 up, 4 in 2024-03-26T04:15:21.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:21 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:15:21.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:21 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:15:21.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:21 smithi016 ceph-mon[29101]: from='osd.3 v1:172.21.15.16:6813/1463789444' entity='osd.3' 2024-03-26T04:15:21.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:21 smithi016 ceph-mon[37215]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:15:21.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:21 smithi016 ceph-mon[37215]: from='osd.3 v1:172.21.15.16:6813/1463789444' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi016", "root=default"]}]': finished 2024-03-26T04:15:21.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:21 smithi016 ceph-mon[37215]: osdmap e24: 4 total, 3 up, 4 in 2024-03-26T04:15:21.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:21 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:15:21.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:21 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:15:21.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:21 smithi016 ceph-mon[37215]: from='osd.3 v1:172.21.15.16:6813/1463789444' entity='osd.3' 2024-03-26T04:15:22.233 DEBUG:teuthology.orchestra.run.smithi183:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi183:vg_nvme/lv_4 2024-03-26T04:15:22.477 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:15:22.510 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:22 smithi183 ceph-mon[33745]: purged_snaps scrub starts 2024-03-26T04:15:22.510 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:22 smithi183 ceph-mon[33745]: purged_snaps scrub ok 2024-03-26T04:15:22.510 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:22 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:15:22.510 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:22 smithi183 ceph-mon[33745]: osd.3 v1:172.21.15.16:6813/1463789444 boot 2024-03-26T04:15:22.510 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:22 smithi183 ceph-mon[33745]: osdmap e25: 4 total, 4 up, 4 in 2024-03-26T04:15:22.510 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:22 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:15:22.510 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:22 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:22.510 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:22 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:22.510 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:22 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:15:22.510 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:22 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:15:22.510 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:22 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:15:22.510 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:22 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:15:22.511 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:22 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:22.511 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:22 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:15:22.511 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:22 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:15:22.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:22 smithi016 ceph-mon[29101]: purged_snaps scrub starts 2024-03-26T04:15:22.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:22 smithi016 ceph-mon[29101]: purged_snaps scrub ok 2024-03-26T04:15:22.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:22 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:15:22.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:22 smithi016 ceph-mon[29101]: osd.3 v1:172.21.15.16:6813/1463789444 boot 2024-03-26T04:15:22.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:22 smithi016 ceph-mon[29101]: osdmap e25: 4 total, 4 up, 4 in 2024-03-26T04:15:22.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:22 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:15:22.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:22 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:22.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:22 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:22.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:22 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:15:22.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:22 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:15:22.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:22 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:15:22.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:22 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:15:22.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:22 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:22.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:22 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:15:22.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:22 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:15:22.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:22 smithi016 ceph-mon[37215]: purged_snaps scrub starts 2024-03-26T04:15:22.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:22 smithi016 ceph-mon[37215]: purged_snaps scrub ok 2024-03-26T04:15:22.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:22 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:15:22.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:22 smithi016 ceph-mon[37215]: osd.3 v1:172.21.15.16:6813/1463789444 boot 2024-03-26T04:15:22.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:22 smithi016 ceph-mon[37215]: osdmap e25: 4 total, 4 up, 4 in 2024-03-26T04:15:22.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:22 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:15:22.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:22 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:22.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:22 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:22.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:22 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:15:22.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:22 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:15:22.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:22 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:15:22.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:22 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:15:22.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:22 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:22.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:22 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:15:22.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:22 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:15:23.453 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:23 smithi183 ceph-mon[33745]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:15:23.454 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:23 smithi183 ceph-mon[33745]: Detected new or changed devices on smithi016 2024-03-26T04:15:23.454 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:23 smithi183 ceph-mon[33745]: Adjusting osd_memory_target on smithi016 to 4017M 2024-03-26T04:15:23.454 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:23 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:23.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:23 smithi016 ceph-mon[29101]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:15:23.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:23 smithi016 ceph-mon[29101]: Detected new or changed devices on smithi016 2024-03-26T04:15:23.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:23 smithi016 ceph-mon[29101]: Adjusting osd_memory_target on smithi016 to 4017M 2024-03-26T04:15:23.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:23 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:23.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:23 smithi016 ceph-mon[37215]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-26T04:15:23.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:23 smithi016 ceph-mon[37215]: Detected new or changed devices on smithi016 2024-03-26T04:15:23.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:23 smithi016 ceph-mon[37215]: Adjusting osd_memory_target on smithi016 to 4017M 2024-03-26T04:15:23.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:23 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:23.968 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:23.967+0000 7fdd5f342700 1 -- 172.21.15.183:0/2885637451 >> v1:172.21.15.183:6789/0 conn(0x7fdd58100d70 legacy=0x7fdd58101190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:15:23.968 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:23.967+0000 7fdd5f342700 1 -- 172.21.15.183:0/2885637451 shutdown_connections 2024-03-26T04:15:23.968 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:23.967+0000 7fdd5f342700 1 -- 172.21.15.183:0/2885637451 >> 172.21.15.183:0/2885637451 conn(0x7fdd580fc310 msgr2=0x7fdd580fe750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:15:23.968 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:23.967+0000 7fdd5f342700 1 -- 172.21.15.183:0/2885637451 shutdown_connections 2024-03-26T04:15:23.969 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:23.967+0000 7fdd5f342700 1 -- 172.21.15.183:0/2885637451 wait complete. 2024-03-26T04:15:23.969 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:23.968+0000 7fdd5f342700 1 Processor -- start 2024-03-26T04:15:23.969 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:23.968+0000 7fdd5f342700 1 -- start start 2024-03-26T04:15:23.971 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:23.968+0000 7fdd5f342700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdd58196be0 con 0x7fdd581051a0 2024-03-26T04:15:23.971 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:23.968+0000 7fdd5f342700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdd581a23d0 con 0x7fdd58108ee0 2024-03-26T04:15:23.971 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:23.968+0000 7fdd5f342700 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdd581a3570 con 0x7fdd58100d70 2024-03-26T04:15:23.971 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:23.968+0000 7fdd5d8df700 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fdd58108ee0 0x7fdd5819fca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:33092/0 (socket says 172.21.15.183:33092) 2024-03-26T04:15:23.971 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:23.968+0000 7fdd5d8df700 1 -- 172.21.15.183:0/1313555953 learned_addr learned my addr 172.21.15.183:0/1313555953 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-26T04:15:23.971 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:23.968+0000 7fdd4e7fc700 1 -- 172.21.15.183:0/1313555953 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3903563657 0 0) 0x7fdd581a23d0 con 0x7fdd58108ee0 2024-03-26T04:15:23.971 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:23.968+0000 7fdd4e7fc700 1 -- 172.21.15.183:0/1313555953 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd440035b0 con 0x7fdd58108ee0 2024-03-26T04:15:23.971 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:23.969+0000 7fdd4e7fc700 1 -- 172.21.15.183:0/1313555953 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3904553915 0 0) 0x7fdd581a3570 con 0x7fdd58100d70 2024-03-26T04:15:23.971 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:23.969+0000 7fdd4e7fc700 1 -- 172.21.15.183:0/1313555953 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd581a23d0 con 0x7fdd58100d70 2024-03-26T04:15:23.971 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:23.969+0000 7fdd4e7fc700 1 -- 172.21.15.183:0/1313555953 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 889225188 0 0) 0x7fdd440035b0 con 0x7fdd58108ee0 2024-03-26T04:15:23.971 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:23.969+0000 7fdd4e7fc700 1 -- 172.21.15.183:0/1313555953 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd581a3570 con 0x7fdd58108ee0 2024-03-26T04:15:23.972 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:23.969+0000 7fdd4e7fc700 1 -- 172.21.15.183:0/1313555953 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fdd50001c10 con 0x7fdd58108ee0 2024-03-26T04:15:23.972 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:23.969+0000 7fdd4e7fc700 1 -- 172.21.15.183:0/1313555953 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2682578119 0 0) 0x7fdd58196be0 con 0x7fdd581051a0 2024-03-26T04:15:23.972 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:23.969+0000 7fdd4e7fc700 1 -- 172.21.15.183:0/1313555953 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd440035b0 con 0x7fdd581051a0 2024-03-26T04:15:23.972 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:23.969+0000 7fdd4e7fc700 1 -- 172.21.15.183:0/1313555953 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 434194059 0 0) 0x7fdd581a3570 con 0x7fdd58108ee0 2024-03-26T04:15:23.972 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:23.969+0000 7fdd4e7fc700 1 -- 172.21.15.183:0/1313555953 >> v1:172.21.15.16:6790/0 conn(0x7fdd58100d70 legacy=0x7fdd58196090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:15:23.972 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:23.969+0000 7fdd4e7fc700 1 -- 172.21.15.183:0/1313555953 >> v1:172.21.15.16:6789/0 conn(0x7fdd581051a0 legacy=0x7fdd5819c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:15:23.972 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:23.969+0000 7fdd4e7fc700 1 -- 172.21.15.183:0/1313555953 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd581a4710 con 0x7fdd58108ee0 2024-03-26T04:15:23.972 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:23.969+0000 7fdd5f342700 1 -- 172.21.15.183:0/1313555953 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd581a3760 con 0x7fdd58108ee0 2024-03-26T04:15:23.972 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:23.969+0000 7fdd5f342700 1 -- 172.21.15.183:0/1313555953 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd581a3c60 con 0x7fdd58108ee0 2024-03-26T04:15:23.972 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:23.969+0000 7fdd4e7fc700 1 -- 172.21.15.183:0/1313555953 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7fdd50003eb0 con 0x7fdd58108ee0 2024-03-26T04:15:23.972 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:23.969+0000 7fdd4e7fc700 1 -- 172.21.15.183:0/1313555953 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fdd50005fb0 con 0x7fdd58108ee0 2024-03-26T04:15:23.972 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:23.970+0000 7fdd5f342700 1 -- 172.21.15.183:0/1313555953 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd3c005150 con 0x7fdd58108ee0 2024-03-26T04:15:23.972 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:23.970+0000 7fdd4e7fc700 1 -- 172.21.15.183:0/1313555953 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 15) v1 ==== 91363+0+0 (unknown 160314185 0 0) 0x7fdd5001c6c0 con 0x7fdd58108ee0 2024-03-26T04:15:23.973 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:23.971+0000 7fdd4e7fc700 1 -- 172.21.15.183:0/1313555953 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(26..26 src has 1..26) v4 ==== 2803+0+0 (unknown 4243153136 0 0) 0x7fdd50087e50 con 0x7fdd58108ee0 2024-03-26T04:15:23.976 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:23.975+0000 7fdd4e7fc700 1 -- 172.21.15.183:0/1313555953 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fdd50052300 con 0x7fdd58108ee0 2024-03-26T04:15:24.178 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:24.178+0000 7fdd5f342700 1 -- 172.21.15.183:0/1313555953 --> v1:172.21.15.16:6800/4042489921 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fdd3c000bc0 con 0x7fdd4406e810 2024-03-26T04:15:24.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:24 smithi183 ceph-mon[33745]: osdmap e26: 4 total, 4 up, 4 in 2024-03-26T04:15:24.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:24 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:15:24.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:24 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:15:24.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:24 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:15:24.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:24 smithi016 ceph-mon[29101]: osdmap e26: 4 total, 4 up, 4 in 2024-03-26T04:15:24.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:24 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:15:24.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:24 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:15:24.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:24 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:15:24.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:24 smithi016 ceph-mon[37215]: osdmap e26: 4 total, 4 up, 4 in 2024-03-26T04:15:24.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:24 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:15:24.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:24 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:15:24.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:24 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:15:25.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:25 smithi016 ceph-mon[29101]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:15:25.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:25 smithi016 ceph-mon[29101]: from='client.24226 v1:172.21.15.183:0/1313555953' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:15:25.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:25 smithi016 ceph-mon[37215]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:15:25.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:25 smithi016 ceph-mon[37215]: from='client.24226 v1:172.21.15.183:0/1313555953' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:15:25.713 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:25 smithi183 ceph-mon[33745]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:15:25.714 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:25 smithi183 ceph-mon[33745]: from='client.24226 v1:172.21.15.183:0/1313555953' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:15:27.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:27 smithi016 ceph-mon[37215]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:15:27.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:27 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.183:0/3141327078' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "750f2953-ae54-49a2-a428-9848615e9ddc"}]: dispatch 2024-03-26T04:15:27.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:27 smithi016 ceph-mon[37215]: from='client.24232 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "750f2953-ae54-49a2-a428-9848615e9ddc"}]: dispatch 2024-03-26T04:15:27.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:27 smithi016 ceph-mon[37215]: from='client.24232 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "750f2953-ae54-49a2-a428-9848615e9ddc"}]': finished 2024-03-26T04:15:27.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:27 smithi016 ceph-mon[37215]: osdmap e27: 5 total, 4 up, 5 in 2024-03-26T04:15:27.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:27 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:15:27.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:27 smithi016 ceph-mon[29101]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:15:27.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:27 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.183:0/3141327078' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "750f2953-ae54-49a2-a428-9848615e9ddc"}]: dispatch 2024-03-26T04:15:27.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:27 smithi016 ceph-mon[29101]: from='client.24232 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "750f2953-ae54-49a2-a428-9848615e9ddc"}]: dispatch 2024-03-26T04:15:27.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:27 smithi016 ceph-mon[29101]: from='client.24232 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "750f2953-ae54-49a2-a428-9848615e9ddc"}]': finished 2024-03-26T04:15:27.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:27 smithi016 ceph-mon[29101]: osdmap e27: 5 total, 4 up, 5 in 2024-03-26T04:15:27.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:27 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:15:27.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:27 smithi183 ceph-mon[33745]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:15:27.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:27 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.183:0/3141327078' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "750f2953-ae54-49a2-a428-9848615e9ddc"}]: dispatch 2024-03-26T04:15:27.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:27 smithi183 ceph-mon[33745]: from='client.24232 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "750f2953-ae54-49a2-a428-9848615e9ddc"}]: dispatch 2024-03-26T04:15:27.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:27 smithi183 ceph-mon[33745]: from='client.24232 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "750f2953-ae54-49a2-a428-9848615e9ddc"}]': finished 2024-03-26T04:15:27.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:27 smithi183 ceph-mon[33745]: osdmap e27: 5 total, 4 up, 5 in 2024-03-26T04:15:27.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:27 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:15:28.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:28 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.183:0/440031778' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:15:28.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:28 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.183:0/440031778' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:15:28.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:28 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.183:0/440031778' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:15:29.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:29 smithi016 ceph-mon[29101]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:15:29.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:29 smithi016 ceph-mon[37215]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:15:29.664 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:29 smithi183 ceph-mon[33745]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:15:31.660 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:31 smithi183 ceph-mon[33745]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:15:31.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:31 smithi016 ceph-mon[29101]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:15:31.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:31 smithi016 ceph-mon[37215]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:15:33.666 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:33 smithi183 ceph-mon[33745]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:15:33.666 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:33 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T04:15:33.666 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:33 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:15:33.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:33 smithi016 ceph-mon[29101]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:15:33.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:33 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T04:15:33.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:33 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:15:33.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:33 smithi016 ceph-mon[37215]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:15:33.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:33 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-26T04:15:33.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:33 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:15:34.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:34 smithi183 ceph-mon[33745]: Deploying daemon osd.4 on smithi183 2024-03-26T04:15:34.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:34 smithi016 ceph-mon[29101]: Deploying daemon osd.4 on smithi183 2024-03-26T04:15:34.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:34 smithi016 ceph-mon[37215]: Deploying daemon osd.4 on smithi183 2024-03-26T04:15:35.628 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:35 smithi183 ceph-mon[33745]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:15:35.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:35 smithi016 ceph-mon[29101]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:15:35.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:35 smithi016 ceph-mon[37215]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:15:37.644 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:37 smithi183 ceph-mon[33745]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:15:37.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:37 smithi016 ceph-mon[29101]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:15:37.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:37 smithi016 ceph-mon[37215]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:15:39.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:39 smithi183 ceph-mon[33745]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:15:39.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:39 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:39.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:39 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:39.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:39 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:15:39.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:39 smithi016 ceph-mon[29101]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:15:39.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:39 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:39.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:39 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:39.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:39 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:15:39.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:39 smithi016 ceph-mon[37215]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:15:39.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:39 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:39.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:39 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:39.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:39 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:15:41.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:40 smithi016 ceph-mon[29101]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:15:41.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:40 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:41.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:40 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:41.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:40 smithi016 ceph-mon[37215]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:15:41.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:40 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:41.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:40 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:41.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:40 smithi183 ceph-mon[33745]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:15:41.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:40 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:41.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:40 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:41.691 INFO:teuthology.orchestra.run.smithi183.stdout:Created osd(s) 4 on host 'smithi183' 2024-03-26T04:15:41.692 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:41.690+0000 7fdd4e7fc700 1 -- 172.21.15.183:0/1313555953 <== mgr.14152 v1:172.21.15.16:6800/4042489921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 1297213289) 0x7fdd3c000bc0 con 0x7fdd4406e810 2024-03-26T04:15:41.693 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:41.692+0000 7fdd5f342700 1 -- 172.21.15.183:0/1313555953 >> v1:172.21.15.16:6800/4042489921 conn(0x7fdd4406e810 legacy=0x7fdd44070cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:15:41.693 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:41.692+0000 7fdd5f342700 1 -- 172.21.15.183:0/1313555953 >> v1:172.21.15.183:6789/0 conn(0x7fdd58108ee0 legacy=0x7fdd5819fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:15:41.693 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:41.692+0000 7fdd5f342700 1 -- 172.21.15.183:0/1313555953 shutdown_connections 2024-03-26T04:15:41.693 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:41.692+0000 7fdd5f342700 1 -- 172.21.15.183:0/1313555953 >> 172.21.15.183:0/1313555953 conn(0x7fdd580fc310 msgr2=0x7fdd581075f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:15:41.693 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:41.693+0000 7fdd5f342700 1 -- 172.21.15.183:0/1313555953 shutdown_connections 2024-03-26T04:15:41.693 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:41.693+0000 7fdd5f342700 1 -- 172.21.15.183:0/1313555953 wait complete. 2024-03-26T04:15:42.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:41 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:15:42.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:41 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:15:42.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:41 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:42.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:41 smithi183 ceph-mon[33745]: from='osd.4 v1:172.21.15.183:6800/879252532' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T04:15:42.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:41 smithi183 ceph-mon[33745]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T04:15:42.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:41 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:42.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:41 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:42.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:41 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:15:42.376 DEBUG:teuthology.orchestra.run.smithi183:osd.4> sudo journalctl -f -n 0 -u ceph-d7d59826-eb26-11ee-95d0-87774f69a715@osd.4.service 2024-03-26T04:15:42.379 INFO:tasks.cephadm:Deploying osd.5 on smithi183 with /dev/vg_nvme/lv_3... 2024-03-26T04:15:42.379 DEBUG:teuthology.orchestra.run.smithi183:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-26T04:15:42.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:41 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:15:42.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:41 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:15:42.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:41 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:42.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:41 smithi016 ceph-mon[29101]: from='osd.4 v1:172.21.15.183:6800/879252532' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T04:15:42.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:41 smithi016 ceph-mon[29101]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T04:15:42.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:41 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:42.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:41 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:42.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:41 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:15:42.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:41 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:15:42.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:41 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:15:42.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:41 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:42.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:41 smithi016 ceph-mon[37215]: from='osd.4 v1:172.21.15.183:6800/879252532' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T04:15:42.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:41 smithi016 ceph-mon[37215]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-26T04:15:42.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:41 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:42.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:41 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:42.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:41 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:15:43.248 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:42 smithi183 ceph-mon[33745]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:15:43.249 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:42 smithi183 ceph-mon[33745]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T04:15:43.249 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:42 smithi183 ceph-mon[33745]: osdmap e28: 5 total, 4 up, 5 in 2024-03-26T04:15:43.249 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:42 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:15:43.249 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:42 smithi183 ceph-mon[33745]: from='osd.4 v1:172.21.15.183:6800/879252532' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-26T04:15:43.249 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:42 smithi183 ceph-mon[33745]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-26T04:15:43.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:42 smithi016 ceph-mon[29101]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:15:43.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:42 smithi016 ceph-mon[29101]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T04:15:43.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:42 smithi016 ceph-mon[29101]: osdmap e28: 5 total, 4 up, 5 in 2024-03-26T04:15:43.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:42 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:15:43.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:42 smithi016 ceph-mon[29101]: from='osd.4 v1:172.21.15.183:6800/879252532' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-26T04:15:43.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:42 smithi016 ceph-mon[29101]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-26T04:15:43.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:42 smithi016 ceph-mon[37215]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:15:43.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:42 smithi016 ceph-mon[37215]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-26T04:15:43.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:42 smithi016 ceph-mon[37215]: osdmap e28: 5 total, 4 up, 5 in 2024-03-26T04:15:43.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:42 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:15:43.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:42 smithi016 ceph-mon[37215]: from='osd.4 v1:172.21.15.183:6800/879252532' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-26T04:15:43.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:42 smithi016 ceph-mon[37215]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-26T04:15:43.540 INFO:journalctl@ceph.osd.4.smithi183.stdout:Mar 26 04:15:43 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-4[38631]: 2024-03-26T04:15:43.435+0000 7fada9cc8700 -1 osd.4 0 waiting for initial osdmap 2024-03-26T04:15:43.541 INFO:journalctl@ceph.osd.4.smithi183.stdout:Mar 26 04:15:43 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-4[38631]: 2024-03-26T04:15:43.441+0000 7fada12b2700 -1 osd.4 29 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T04:15:43.586 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:15:44.271 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:43 smithi183 ceph-mon[33745]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-03-26T04:15:44.271 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:43 smithi183 ceph-mon[33745]: osdmap e29: 5 total, 4 up, 5 in 2024-03-26T04:15:44.271 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:43 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:15:44.271 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:43 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:15:44.271 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:43 smithi183 ceph-mon[33745]: from='osd.4 ' entity='osd.4' 2024-03-26T04:15:44.271 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:43 smithi183 ceph-mon[33745]: osd.4 v1:172.21.15.183:6800/879252532 boot 2024-03-26T04:15:44.271 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:43 smithi183 ceph-mon[33745]: osdmap e30: 5 total, 5 up, 5 in 2024-03-26T04:15:44.272 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:43 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:15:44.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:43 smithi016 ceph-mon[29101]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-03-26T04:15:44.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:43 smithi016 ceph-mon[29101]: osdmap e29: 5 total, 4 up, 5 in 2024-03-26T04:15:44.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:43 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:15:44.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:43 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:15:44.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:43 smithi016 ceph-mon[29101]: from='osd.4 ' entity='osd.4' 2024-03-26T04:15:44.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:43 smithi016 ceph-mon[29101]: osd.4 v1:172.21.15.183:6800/879252532 boot 2024-03-26T04:15:44.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:43 smithi016 ceph-mon[29101]: osdmap e30: 5 total, 5 up, 5 in 2024-03-26T04:15:44.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:43 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:15:44.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:43 smithi016 ceph-mon[37215]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-03-26T04:15:44.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:43 smithi016 ceph-mon[37215]: osdmap e29: 5 total, 4 up, 5 in 2024-03-26T04:15:44.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:43 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:15:44.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:43 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:15:44.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:43 smithi016 ceph-mon[37215]: from='osd.4 ' entity='osd.4' 2024-03-26T04:15:44.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:43 smithi016 ceph-mon[37215]: osd.4 v1:172.21.15.183:6800/879252532 boot 2024-03-26T04:15:44.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:43 smithi016 ceph-mon[37215]: osdmap e30: 5 total, 5 up, 5 in 2024-03-26T04:15:44.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:43 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:15:45.143 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:44 smithi183 ceph-mon[33745]: purged_snaps scrub starts 2024-03-26T04:15:45.143 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:44 smithi183 ceph-mon[33745]: purged_snaps scrub ok 2024-03-26T04:15:45.143 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:44 smithi183 ceph-mon[33745]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:15:45.144 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:44 smithi183 ceph-mon[33745]: osdmap e31: 5 total, 5 up, 5 in 2024-03-26T04:15:45.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:44 smithi016 ceph-mon[29101]: purged_snaps scrub starts 2024-03-26T04:15:45.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:44 smithi016 ceph-mon[29101]: purged_snaps scrub ok 2024-03-26T04:15:45.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:44 smithi016 ceph-mon[29101]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:15:45.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:44 smithi016 ceph-mon[29101]: osdmap e31: 5 total, 5 up, 5 in 2024-03-26T04:15:45.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:44 smithi016 ceph-mon[37215]: purged_snaps scrub starts 2024-03-26T04:15:45.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:44 smithi016 ceph-mon[37215]: purged_snaps scrub ok 2024-03-26T04:15:45.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:44 smithi016 ceph-mon[37215]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 116 MiB used, 357 GiB / 358 GiB avail 2024-03-26T04:15:45.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:44 smithi016 ceph-mon[37215]: osdmap e31: 5 total, 5 up, 5 in 2024-03-26T04:15:45.904 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-26T04:15:45.904 INFO:teuthology.orchestra.run.smithi183.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-26T04:15:45.904 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10+0 records in 2024-03-26T04:15:45.905 INFO:teuthology.orchestra.run.smithi183.stderr:10+0 records out 2024-03-26T04:15:45.905 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0141087 s, 743 MB/s 2024-03-26T04:15:45.905 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping successful for: 2024-03-26T04:15:46.712 DEBUG:teuthology.orchestra.run.smithi183:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi183:vg_nvme/lv_3 2024-03-26T04:15:46.947 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:15:47.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:46 smithi183 ceph-mon[33745]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 535 MiB used, 446 GiB / 447 GiB avail 2024-03-26T04:15:47.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:46 smithi183 ceph-mon[33745]: osdmap e32: 5 total, 5 up, 5 in 2024-03-26T04:15:47.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:46 smithi183 ceph-mon[33745]: Detected new or changed devices on smithi183 2024-03-26T04:15:47.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:46 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:47.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:46 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:47.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:46 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:15:47.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:46 smithi183 ceph-mon[33745]: Adjusting osd_memory_target on smithi183 to 17088M 2024-03-26T04:15:47.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:46 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:47.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:46 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:15:47.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:46 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:15:47.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:46 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:47.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:46 smithi016 ceph-mon[29101]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 535 MiB used, 446 GiB / 447 GiB avail 2024-03-26T04:15:47.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:46 smithi016 ceph-mon[29101]: osdmap e32: 5 total, 5 up, 5 in 2024-03-26T04:15:47.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:46 smithi016 ceph-mon[29101]: Detected new or changed devices on smithi183 2024-03-26T04:15:47.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:46 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:47.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:46 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:47.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:46 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:15:47.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:46 smithi016 ceph-mon[29101]: Adjusting osd_memory_target on smithi183 to 17088M 2024-03-26T04:15:47.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:46 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:47.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:46 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:15:47.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:46 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:15:47.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:46 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:47.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:46 smithi016 ceph-mon[37215]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 535 MiB used, 446 GiB / 447 GiB avail 2024-03-26T04:15:47.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:46 smithi016 ceph-mon[37215]: osdmap e32: 5 total, 5 up, 5 in 2024-03-26T04:15:47.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:46 smithi016 ceph-mon[37215]: Detected new or changed devices on smithi183 2024-03-26T04:15:47.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:46 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:47.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:46 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:47.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:46 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:15:47.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:46 smithi016 ceph-mon[37215]: Adjusting osd_memory_target on smithi183 to 17088M 2024-03-26T04:15:47.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:46 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:47.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:46 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:15:47.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:46 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:15:47.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:46 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:15:48.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:47 smithi183 ceph-mon[33745]: osdmap e33: 5 total, 5 up, 5 in 2024-03-26T04:15:48.334 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:48.331+0000 7f7a48c6c700 1 -- 172.21.15.183:0/819226501 >> v1:172.21.15.183:6789/0 conn(0x7f7a44100d80 legacy=0x7f7a441011a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:15:48.335 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:48.331+0000 7f7a48c6c700 1 -- 172.21.15.183:0/819226501 shutdown_connections 2024-03-26T04:15:48.335 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:48.331+0000 7f7a48c6c700 1 -- 172.21.15.183:0/819226501 >> 172.21.15.183:0/819226501 conn(0x7f7a440fc340 msgr2=0x7f7a440fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:15:48.335 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:48.331+0000 7f7a48c6c700 1 -- 172.21.15.183:0/819226501 shutdown_connections 2024-03-26T04:15:48.335 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:48.331+0000 7f7a48c6c700 1 -- 172.21.15.183:0/819226501 wait complete. 2024-03-26T04:15:48.335 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:48.332+0000 7f7a48c6c700 1 Processor -- start 2024-03-26T04:15:48.335 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:48.332+0000 7f7a48c6c700 1 -- start start 2024-03-26T04:15:48.335 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:48.332+0000 7f7a48c6c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7a441a4530 con 0x7f7a44108ef0 2024-03-26T04:15:48.335 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:48.332+0000 7f7a48c6c700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7a441a5730 con 0x7f7a44100d80 2024-03-26T04:15:48.335 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:48.332+0000 7f7a48c6c700 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7a441a6930 con 0x7f7a441051b0 2024-03-26T04:15:48.335 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:48.332+0000 7f7a4259c700 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f7a44100d80 0x7f7a44107080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:58112/0 (socket says 172.21.15.183:58112) 2024-03-26T04:15:48.336 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:48.332+0000 7f7a4259c700 1 -- 172.21.15.183:0/3613702794 learned_addr learned my addr 172.21.15.183:0/3613702794 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-26T04:15:48.336 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:48.333+0000 7f7a337fe700 1 -- 172.21.15.183:0/3613702794 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2294541794 0 0) 0x7f7a441a5730 con 0x7f7a44100d80 2024-03-26T04:15:48.336 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:48.333+0000 7f7a337fe700 1 -- 172.21.15.183:0/3613702794 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a2c0035b0 con 0x7f7a44100d80 2024-03-26T04:15:48.336 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:48.333+0000 7f7a337fe700 1 -- 172.21.15.183:0/3613702794 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2560179114 0 0) 0x7f7a441a4530 con 0x7f7a44108ef0 2024-03-26T04:15:48.336 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:48.333+0000 7f7a337fe700 1 -- 172.21.15.183:0/3613702794 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a441a5730 con 0x7f7a44108ef0 2024-03-26T04:15:48.336 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:48.333+0000 7f7a337fe700 1 -- 172.21.15.183:0/3613702794 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2650015128 0 0) 0x7f7a441a6930 con 0x7f7a441051b0 2024-03-26T04:15:48.336 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:48.333+0000 7f7a337fe700 1 -- 172.21.15.183:0/3613702794 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a441a4530 con 0x7f7a441051b0 2024-03-26T04:15:48.336 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:48.333+0000 7f7a337fe700 1 -- 172.21.15.183:0/3613702794 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4121695438 0 0) 0x7f7a2c0035b0 con 0x7f7a44100d80 2024-03-26T04:15:48.336 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:48.333+0000 7f7a337fe700 1 -- 172.21.15.183:0/3613702794 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a441a6930 con 0x7f7a44100d80 2024-03-26T04:15:48.336 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:48.333+0000 7f7a337fe700 1 -- 172.21.15.183:0/3613702794 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f7a34001370 con 0x7f7a44100d80 2024-03-26T04:15:48.336 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:48.333+0000 7f7a337fe700 1 -- 172.21.15.183:0/3613702794 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 817209767 0 0) 0x7f7a441a6930 con 0x7f7a44100d80 2024-03-26T04:15:48.336 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:48.333+0000 7f7a337fe700 1 -- 172.21.15.183:0/3613702794 >> v1:172.21.15.16:6790/0 conn(0x7f7a441051b0 legacy=0x7f7a4419f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:15:48.337 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:48.333+0000 7f7a337fe700 1 -- 172.21.15.183:0/3613702794 >> v1:172.21.15.16:6789/0 conn(0x7f7a44108ef0 legacy=0x7f7a441a2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:15:48.337 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:48.333+0000 7f7a337fe700 1 -- 172.21.15.183:0/3613702794 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a441a7b30 con 0x7f7a44100d80 2024-03-26T04:15:48.337 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:48.333+0000 7f7a48c6c700 1 -- 172.21.15.183:0/3613702794 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a441a4720 con 0x7f7a44100d80 2024-03-26T04:15:48.337 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:48.333+0000 7f7a48c6c700 1 -- 172.21.15.183:0/3613702794 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a441a4c20 con 0x7f7a44100d80 2024-03-26T04:15:48.337 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:48.333+0000 7f7a337fe700 1 -- 172.21.15.183:0/3613702794 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f7a34001840 con 0x7f7a44100d80 2024-03-26T04:15:48.337 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:48.333+0000 7f7a337fe700 1 -- 172.21.15.183:0/3613702794 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f7a34004ee0 con 0x7f7a44100d80 2024-03-26T04:15:48.337 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:48.334+0000 7f7a48c6c700 1 -- 172.21.15.183:0/3613702794 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a24005150 con 0x7f7a44100d80 2024-03-26T04:15:48.338 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:48.335+0000 7f7a337fe700 1 -- 172.21.15.183:0/3613702794 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 15) v1 ==== 91363+0+0 (unknown 160314185 0 0) 0x7f7a34003890 con 0x7f7a44100d80 2024-03-26T04:15:48.338 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:48.335+0000 7f7a337fe700 1 -- 172.21.15.183:0/3613702794 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(34..34 src has 1..34) v4 ==== 3205+0+0 (unknown 2157099195 0 0) 0x7f7a34087df0 con 0x7f7a44100d80 2024-03-26T04:15:48.340 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:48.340+0000 7f7a337fe700 1 -- 172.21.15.183:0/3613702794 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f7a34052120 con 0x7f7a44100d80 2024-03-26T04:15:48.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:47 smithi016 ceph-mon[37215]: osdmap e33: 5 total, 5 up, 5 in 2024-03-26T04:15:48.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:47 smithi016 ceph-mon[29101]: osdmap e33: 5 total, 5 up, 5 in 2024-03-26T04:15:48.542 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:15:48.542+0000 7f7a48c6c700 1 -- 172.21.15.183:0/3613702794 --> v1:172.21.15.16:6800/4042489921 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f7a24000bc0 con 0x7f7a2c06e8c0 2024-03-26T04:15:49.280 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:48 smithi183 ceph-mon[33745]: pgmap v115: 1 pgs: 1 peering; 577 KiB data, 535 MiB used, 446 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-26T04:15:49.280 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:48 smithi183 ceph-mon[33745]: osdmap e34: 5 total, 5 up, 5 in 2024-03-26T04:15:49.280 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:48 smithi183 ceph-mon[33745]: from='client.24253 v1:172.21.15.183:0/3613702794' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:15:49.280 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:48 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:15:49.280 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:48 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:15:49.280 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:48 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:15:49.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:48 smithi016 ceph-mon[29101]: pgmap v115: 1 pgs: 1 peering; 577 KiB data, 535 MiB used, 446 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-26T04:15:49.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:48 smithi016 ceph-mon[29101]: osdmap e34: 5 total, 5 up, 5 in 2024-03-26T04:15:49.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:48 smithi016 ceph-mon[29101]: from='client.24253 v1:172.21.15.183:0/3613702794' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:15:49.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:48 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:15:49.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:48 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:15:49.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:48 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:15:49.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:48 smithi016 ceph-mon[37215]: pgmap v115: 1 pgs: 1 peering; 577 KiB data, 535 MiB used, 446 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-26T04:15:49.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:48 smithi016 ceph-mon[37215]: osdmap e34: 5 total, 5 up, 5 in 2024-03-26T04:15:49.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:48 smithi016 ceph-mon[37215]: from='client.24253 v1:172.21.15.183:0/3613702794' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:15:49.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:48 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:15:49.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:48 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:15:49.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:48 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:15:51.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:50 smithi183 ceph-mon[33745]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 144 MiB used, 447 GiB / 447 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-03-26T04:15:51.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:50 smithi016 ceph-mon[37215]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 144 MiB used, 447 GiB / 447 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-03-26T04:15:51.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:50 smithi016 ceph-mon[29101]: pgmap v117: 1 pgs: 1 peering; 577 KiB data, 144 MiB used, 447 GiB / 447 GiB avail; 120 KiB/s, 0 objects/s recovering 2024-03-26T04:15:52.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:51 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.183:0/3360749074' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0bbc822d-07fc-45c7-809d-a77e2cba6428"}]: dispatch 2024-03-26T04:15:52.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:51 smithi183 ceph-mon[33745]: from='client.24259 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0bbc822d-07fc-45c7-809d-a77e2cba6428"}]: dispatch 2024-03-26T04:15:52.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:51 smithi183 ceph-mon[33745]: from='client.24259 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0bbc822d-07fc-45c7-809d-a77e2cba6428"}]': finished 2024-03-26T04:15:52.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:51 smithi183 ceph-mon[33745]: osdmap e35: 6 total, 5 up, 6 in 2024-03-26T04:15:52.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:51 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:15:52.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:51 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.183:0/1980005762' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:15:52.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:51 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.183:0/3360749074' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0bbc822d-07fc-45c7-809d-a77e2cba6428"}]: dispatch 2024-03-26T04:15:52.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:51 smithi016 ceph-mon[37215]: from='client.24259 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0bbc822d-07fc-45c7-809d-a77e2cba6428"}]: dispatch 2024-03-26T04:15:52.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:51 smithi016 ceph-mon[37215]: from='client.24259 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0bbc822d-07fc-45c7-809d-a77e2cba6428"}]': finished 2024-03-26T04:15:52.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:51 smithi016 ceph-mon[37215]: osdmap e35: 6 total, 5 up, 6 in 2024-03-26T04:15:52.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:51 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:15:52.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:51 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.183:0/1980005762' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:15:52.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:51 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.183:0/3360749074' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0bbc822d-07fc-45c7-809d-a77e2cba6428"}]: dispatch 2024-03-26T04:15:52.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:51 smithi016 ceph-mon[29101]: from='client.24259 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0bbc822d-07fc-45c7-809d-a77e2cba6428"}]: dispatch 2024-03-26T04:15:52.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:51 smithi016 ceph-mon[29101]: from='client.24259 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0bbc822d-07fc-45c7-809d-a77e2cba6428"}]': finished 2024-03-26T04:15:52.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:51 smithi016 ceph-mon[29101]: osdmap e35: 6 total, 5 up, 6 in 2024-03-26T04:15:52.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:51 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:15:52.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:51 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.183:0/1980005762' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:15:53.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:53 smithi016 ceph-mon[37215]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 144 MiB used, 447 GiB / 447 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-03-26T04:15:53.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:53 smithi016 ceph-mon[29101]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 144 MiB used, 447 GiB / 447 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-03-26T04:15:53.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:53 smithi183 ceph-mon[33745]: pgmap v119: 1 pgs: 1 peering; 577 KiB data, 144 MiB used, 447 GiB / 447 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-03-26T04:15:55.336 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:55 smithi183 ceph-mon[33745]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-03-26T04:15:55.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:55 smithi016 ceph-mon[37215]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-03-26T04:15:55.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:55 smithi016 ceph-mon[29101]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail; 85 KiB/s, 0 objects/s recovering 2024-03-26T04:15:57.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:57 smithi016 ceph-mon[37215]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:15:57.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:57 smithi016 ceph-mon[29101]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:15:57.503 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:57 smithi183 ceph-mon[33745]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:15:58.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:58 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T04:15:58.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:58 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:15:58.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:58 smithi016 ceph-mon[37215]: Deploying daemon osd.5 on smithi183 2024-03-26T04:15:58.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:58 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T04:15:58.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:58 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:15:58.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:58 smithi016 ceph-mon[29101]: Deploying daemon osd.5 on smithi183 2024-03-26T04:15:58.464 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:58 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-26T04:15:58.465 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:58 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:15:58.465 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:58 smithi183 ceph-mon[33745]: Deploying daemon osd.5 on smithi183 2024-03-26T04:15:59.399 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:15:59 smithi183 ceph-mon[33745]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:15:59.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:15:59 smithi016 ceph-mon[37215]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:15:59.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:15:59 smithi016 ceph-mon[29101]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:16:01.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:01 smithi183 ceph-mon[33745]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:16:01.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:01 smithi016 ceph-mon[29101]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:16:01.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:01 smithi016 ceph-mon[37215]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:16:03.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:03 smithi016 ceph-mon[37215]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:16:03.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:03 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:03.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:03 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:03.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:03 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:16:03.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:03 smithi016 ceph-mon[29101]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:16:03.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:03 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:03.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:03 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:03.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:03 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:16:03.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:03 smithi183 ceph-mon[33745]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:16:03.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:03 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:03.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:03 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:03.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:03 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:16:05.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:05 smithi016 ceph-mon[29101]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:16:05.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:05 smithi016 ceph-mon[37215]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:16:05.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:05 smithi183 ceph-mon[33745]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:16:05.854 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:05.853+0000 7f7a337fe700 1 -- 172.21.15.183:0/3613702794 <== mgr.14152 v1:172.21.15.16:6800/4042489921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 289671645) 0x7f7a24000bc0 con 0x7f7a2c06e8c0 2024-03-26T04:16:05.855 INFO:teuthology.orchestra.run.smithi183.stdout:Created osd(s) 5 on host 'smithi183' 2024-03-26T04:16:05.856 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:05.855+0000 7f7a48c6c700 1 -- 172.21.15.183:0/3613702794 >> v1:172.21.15.16:6800/4042489921 conn(0x7f7a2c06e8c0 legacy=0x7f7a2c070d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:16:05.856 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:05.856+0000 7f7a48c6c700 1 -- 172.21.15.183:0/3613702794 >> v1:172.21.15.183:6789/0 conn(0x7f7a44100d80 legacy=0x7f7a44107080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:16:05.856 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:05.856+0000 7f7a48c6c700 1 -- 172.21.15.183:0/3613702794 shutdown_connections 2024-03-26T04:16:05.856 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:05.856+0000 7f7a48c6c700 1 -- 172.21.15.183:0/3613702794 >> 172.21.15.183:0/3613702794 conn(0x7f7a440fc340 msgr2=0x7f7a4406acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:16:05.856 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:05.856+0000 7f7a48c6c700 1 -- 172.21.15.183:0/3613702794 shutdown_connections 2024-03-26T04:16:05.856 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:05.856+0000 7f7a48c6c700 1 -- 172.21.15.183:0/3613702794 wait complete. 2024-03-26T04:16:06.446 DEBUG:teuthology.orchestra.run.smithi183:osd.5> sudo journalctl -f -n 0 -u ceph-d7d59826-eb26-11ee-95d0-87774f69a715@osd.5.service 2024-03-26T04:16:06.449 INFO:tasks.cephadm:Deploying osd.6 on smithi183 with /dev/vg_nvme/lv_2... 2024-03-26T04:16:06.449 DEBUG:teuthology.orchestra.run.smithi183:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-26T04:16:06.831 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:16:07.106 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:06 smithi183 ceph-mon[33745]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:16:07.106 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:06 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:07.106 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:06 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:07.106 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:06 smithi183 ceph-mon[33745]: from='osd.5 v1:172.21.15.183:6804/3356204259' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T04:16:07.106 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:06 smithi183 ceph-mon[33745]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T04:16:07.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:06 smithi016 ceph-mon[29101]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:16:07.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:06 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:07.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:06 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:07.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:06 smithi016 ceph-mon[29101]: from='osd.5 v1:172.21.15.183:6804/3356204259' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T04:16:07.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:06 smithi016 ceph-mon[29101]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T04:16:07.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:06 smithi016 ceph-mon[37215]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:16:07.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:06 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:07.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:06 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:07.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:06 smithi016 ceph-mon[37215]: from='osd.5 v1:172.21.15.183:6804/3356204259' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T04:16:07.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:06 smithi016 ceph-mon[37215]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-26T04:16:07.967 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:07 smithi183 ceph-mon[33745]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T04:16:07.967 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:07 smithi183 ceph-mon[33745]: osdmap e36: 6 total, 5 up, 6 in 2024-03-26T04:16:07.967 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:07 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:16:07.967 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:07 smithi183 ceph-mon[33745]: from='osd.5 v1:172.21.15.183:6804/3356204259' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-26T04:16:07.967 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:07 smithi183 ceph-mon[33745]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-26T04:16:07.967 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:07 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:07.967 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:07 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:07.968 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 26 04:16:07 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-5[44571]: 2024-03-26T04:16:07.860+0000 7fd007d71700 -1 osd.5 0 waiting for initial osdmap 2024-03-26T04:16:07.968 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 26 04:16:07 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-5[44571]: 2024-03-26T04:16:07.865+0000 7fd003b64700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T04:16:08.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:07 smithi016 ceph-mon[37215]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T04:16:08.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:07 smithi016 ceph-mon[37215]: osdmap e36: 6 total, 5 up, 6 in 2024-03-26T04:16:08.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:07 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:16:08.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:07 smithi016 ceph-mon[37215]: from='osd.5 v1:172.21.15.183:6804/3356204259' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-26T04:16:08.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:07 smithi016 ceph-mon[37215]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-26T04:16:08.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:07 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:08.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:07 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:08.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:07 smithi016 ceph-mon[29101]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-26T04:16:08.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:07 smithi016 ceph-mon[29101]: osdmap e36: 6 total, 5 up, 6 in 2024-03-26T04:16:08.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:07 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:16:08.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:07 smithi016 ceph-mon[29101]: from='osd.5 v1:172.21.15.183:6804/3356204259' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-26T04:16:08.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:07 smithi016 ceph-mon[29101]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-26T04:16:08.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:07 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:08.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:07 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:08.712 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-26T04:16:08.712 INFO:teuthology.orchestra.run.smithi183.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-26T04:16:08.712 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10+0 records in 2024-03-26T04:16:08.712 INFO:teuthology.orchestra.run.smithi183.stderr:10+0 records out 2024-03-26T04:16:08.712 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0164763 s, 636 MB/s 2024-03-26T04:16:08.712 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping successful for: 2024-03-26T04:16:09.146 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:08 smithi183 ceph-mon[33745]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:16:09.146 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:08 smithi183 ceph-mon[33745]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-03-26T04:16:09.146 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:08 smithi183 ceph-mon[33745]: osdmap e37: 6 total, 5 up, 6 in 2024-03-26T04:16:09.146 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:08 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:16:09.146 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:08 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:16:09.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:08 smithi016 ceph-mon[29101]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:16:09.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:08 smithi016 ceph-mon[29101]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-03-26T04:16:09.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:08 smithi016 ceph-mon[29101]: osdmap e37: 6 total, 5 up, 6 in 2024-03-26T04:16:09.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:08 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:16:09.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:08 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:16:09.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:08 smithi016 ceph-mon[37215]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:16:09.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:08 smithi016 ceph-mon[37215]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-03-26T04:16:09.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:08 smithi016 ceph-mon[37215]: osdmap e37: 6 total, 5 up, 6 in 2024-03-26T04:16:09.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:08 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:16:09.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:08 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:16:10.058 DEBUG:teuthology.orchestra.run.smithi183:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi183:vg_nvme/lv_2 2024-03-26T04:16:10.148 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:09 smithi183 ceph-mon[33745]: purged_snaps scrub starts 2024-03-26T04:16:10.148 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:09 smithi183 ceph-mon[33745]: purged_snaps scrub ok 2024-03-26T04:16:10.148 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:09 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:16:10.148 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:09 smithi183 ceph-mon[33745]: osd.5 v1:172.21.15.183:6804/3356204259 boot 2024-03-26T04:16:10.148 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:09 smithi183 ceph-mon[33745]: osdmap e38: 6 total, 6 up, 6 in 2024-03-26T04:16:10.148 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:09 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:16:10.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:09 smithi016 ceph-mon[29101]: purged_snaps scrub starts 2024-03-26T04:16:10.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:09 smithi016 ceph-mon[29101]: purged_snaps scrub ok 2024-03-26T04:16:10.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:09 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:16:10.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:09 smithi016 ceph-mon[29101]: osd.5 v1:172.21.15.183:6804/3356204259 boot 2024-03-26T04:16:10.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:09 smithi016 ceph-mon[29101]: osdmap e38: 6 total, 6 up, 6 in 2024-03-26T04:16:10.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:09 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:16:10.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:09 smithi016 ceph-mon[37215]: purged_snaps scrub starts 2024-03-26T04:16:10.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:09 smithi016 ceph-mon[37215]: purged_snaps scrub ok 2024-03-26T04:16:10.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:09 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:16:10.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:09 smithi016 ceph-mon[37215]: osd.5 v1:172.21.15.183:6804/3356204259 boot 2024-03-26T04:16:10.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:09 smithi016 ceph-mon[37215]: osdmap e38: 6 total, 6 up, 6 in 2024-03-26T04:16:10.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:09 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:16:10.565 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:16:11.014 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:10 smithi183 ceph-mon[33745]: pgmap v131: 1 pgs: 1 unknown; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:16:11.014 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:10 smithi183 ceph-mon[33745]: osdmap e39: 6 total, 6 up, 6 in 2024-03-26T04:16:11.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:10 smithi016 ceph-mon[37215]: pgmap v131: 1 pgs: 1 unknown; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:16:11.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:10 smithi016 ceph-mon[37215]: osdmap e39: 6 total, 6 up, 6 in 2024-03-26T04:16:11.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:10 smithi016 ceph-mon[29101]: pgmap v131: 1 pgs: 1 unknown; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-26T04:16:11.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:10 smithi016 ceph-mon[29101]: osdmap e39: 6 total, 6 up, 6 in 2024-03-26T04:16:12.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:11 smithi183 ceph-mon[33745]: osdmap e40: 6 total, 6 up, 6 in 2024-03-26T04:16:12.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:11 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:12.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:11 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:12.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:11 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:16:12.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:11 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:16:12.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:11 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:12.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:11 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:16:12.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:11 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:16:12.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:11 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:12.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:11 smithi016 ceph-mon[29101]: osdmap e40: 6 total, 6 up, 6 in 2024-03-26T04:16:12.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:11 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:12.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:11 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:12.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:11 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:16:12.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:11 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:16:12.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:11 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:12.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:11 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:16:12.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:11 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:16:12.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:11 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:12.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:11 smithi016 ceph-mon[37215]: osdmap e40: 6 total, 6 up, 6 in 2024-03-26T04:16:12.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:11 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:12.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:11 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:12.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:11 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:16:12.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:11 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:16:12.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:11 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:12.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:11 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:16:12.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:11 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:16:12.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:11 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:12.648 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:12.646+0000 7fd8c91c4700 1 -- 172.21.15.183:0/2244584259 >> v1:172.21.15.183:6789/0 conn(0x7fd8c4103490 legacy=0x7fd8c41038b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:16:12.648 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:12.646+0000 7fd8c91c4700 1 -- 172.21.15.183:0/2244584259 shutdown_connections 2024-03-26T04:16:12.648 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:12.646+0000 7fd8c91c4700 1 -- 172.21.15.183:0/2244584259 >> 172.21.15.183:0/2244584259 conn(0x7fd8c40fbaf0 msgr2=0x7fd8c40fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:16:12.648 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:12.646+0000 7fd8c91c4700 1 -- 172.21.15.183:0/2244584259 shutdown_connections 2024-03-26T04:16:12.649 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:12.646+0000 7fd8c91c4700 1 -- 172.21.15.183:0/2244584259 wait complete. 2024-03-26T04:16:12.649 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:12.647+0000 7fd8c91c4700 1 Processor -- start 2024-03-26T04:16:12.649 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:12.647+0000 7fd8c91c4700 1 -- start start 2024-03-26T04:16:12.649 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:12.647+0000 7fd8c91c4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd8c41a4570 con 0x7fd8c4106460 2024-03-26T04:16:12.649 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:12.647+0000 7fd8c91c4700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd8c41a5770 con 0x7fd8c4109fd0 2024-03-26T04:16:12.649 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:12.647+0000 7fd8c91c4700 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd8c41a6970 con 0x7fd8c4103490 2024-03-26T04:16:12.649 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:12.647+0000 7fd8c359e700 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fd8c4109fd0 0x7fd8c41a2c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:57062/0 (socket says 172.21.15.183:57062) 2024-03-26T04:16:12.649 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:12.647+0000 7fd8c359e700 1 -- 172.21.15.183:0/4136821576 learned_addr learned my addr 172.21.15.183:0/4136821576 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-26T04:16:12.649 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:12.648+0000 7fd8abfff700 1 -- 172.21.15.183:0/4136821576 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1315779428 0 0) 0x7fd8c41a5770 con 0x7fd8c4109fd0 2024-03-26T04:16:12.649 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:12.648+0000 7fd8abfff700 1 -- 172.21.15.183:0/4136821576 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8ac0035b0 con 0x7fd8c4109fd0 2024-03-26T04:16:12.649 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:12.648+0000 7fd8abfff700 1 -- 172.21.15.183:0/4136821576 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1608956120 0 0) 0x7fd8c41a6970 con 0x7fd8c4103490 2024-03-26T04:16:12.649 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:12.648+0000 7fd8abfff700 1 -- 172.21.15.183:0/4136821576 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8c41a5770 con 0x7fd8c4103490 2024-03-26T04:16:12.650 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:12.648+0000 7fd8abfff700 1 -- 172.21.15.183:0/4136821576 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2401018013 0 0) 0x7fd8c41a4570 con 0x7fd8c4106460 2024-03-26T04:16:12.650 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:12.648+0000 7fd8abfff700 1 -- 172.21.15.183:0/4136821576 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8c41a6970 con 0x7fd8c4106460 2024-03-26T04:16:12.650 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:12.648+0000 7fd8abfff700 1 -- 172.21.15.183:0/4136821576 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4198277849 0 0) 0x7fd8ac0035b0 con 0x7fd8c4109fd0 2024-03-26T04:16:12.650 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:12.648+0000 7fd8abfff700 1 -- 172.21.15.183:0/4136821576 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8c41a4570 con 0x7fd8c4109fd0 2024-03-26T04:16:12.650 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:12.648+0000 7fd8abfff700 1 -- 172.21.15.183:0/4136821576 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fd8bc001ba0 con 0x7fd8c4109fd0 2024-03-26T04:16:12.650 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:12.648+0000 7fd8abfff700 1 -- 172.21.15.183:0/4136821576 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 970831117 0 0) 0x7fd8c41a5770 con 0x7fd8c4103490 2024-03-26T04:16:12.650 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:12.648+0000 7fd8abfff700 1 -- 172.21.15.183:0/4136821576 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8ac0035b0 con 0x7fd8c4103490 2024-03-26T04:16:12.650 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:12.648+0000 7fd8abfff700 1 -- 172.21.15.183:0/4136821576 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fd8b4001370 con 0x7fd8c4103490 2024-03-26T04:16:12.650 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:12.648+0000 7fd8abfff700 1 -- 172.21.15.183:0/4136821576 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 245233407 0 0) 0x7fd8c41a6970 con 0x7fd8c4106460 2024-03-26T04:16:12.650 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:12.648+0000 7fd8abfff700 1 -- 172.21.15.183:0/4136821576 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8c41a5770 con 0x7fd8c4106460 2024-03-26T04:16:12.650 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:12.648+0000 7fd8abfff700 1 -- 172.21.15.183:0/4136821576 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3711393114 0 0) 0x7fd8c41a4570 con 0x7fd8c4109fd0 2024-03-26T04:16:12.650 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:12.648+0000 7fd8abfff700 1 -- 172.21.15.183:0/4136821576 >> v1:172.21.15.16:6790/0 conn(0x7fd8c4103490 legacy=0x7fd8c4102ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:16:12.650 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:12.648+0000 7fd8abfff700 1 -- 172.21.15.183:0/4136821576 >> v1:172.21.15.16:6789/0 conn(0x7fd8c4106460 legacy=0x7fd8c419f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:16:12.651 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:12.648+0000 7fd8abfff700 1 -- 172.21.15.183:0/4136821576 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8c41a7b70 con 0x7fd8c4109fd0 2024-03-26T04:16:12.651 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:12.648+0000 7fd8c91c4700 1 -- 172.21.15.183:0/4136821576 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8c41a6b60 con 0x7fd8c4109fd0 2024-03-26T04:16:12.651 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:12.648+0000 7fd8c91c4700 1 -- 172.21.15.183:0/4136821576 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8c41a7060 con 0x7fd8c4109fd0 2024-03-26T04:16:12.651 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:12.648+0000 7fd8abfff700 1 -- 172.21.15.183:0/4136821576 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7fd8bc003eb0 con 0x7fd8c4109fd0 2024-03-26T04:16:12.651 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:12.648+0000 7fd8abfff700 1 -- 172.21.15.183:0/4136821576 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fd8bc005f30 con 0x7fd8c4109fd0 2024-03-26T04:16:12.651 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:12.649+0000 7fd8c91c4700 1 -- 172.21.15.183:0/4136821576 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8b0005150 con 0x7fd8c4109fd0 2024-03-26T04:16:12.651 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:12.650+0000 7fd8abfff700 1 -- 172.21.15.183:0/4136821576 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 15) v1 ==== 91363+0+0 (unknown 160314185 0 0) 0x7fd8bc01c600 con 0x7fd8c4109fd0 2024-03-26T04:16:12.652 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:12.650+0000 7fd8abfff700 1 -- 172.21.15.183:0/4136821576 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(41..41 src has 1..41) v4 ==== 3497+0+0 (unknown 1010037731 0 0) 0x7fd8bc088090 con 0x7fd8c4109fd0 2024-03-26T04:16:12.654 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:12.655+0000 7fd8abfff700 1 -- 172.21.15.183:0/4136821576 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fd8bc0522a0 con 0x7fd8c4109fd0 2024-03-26T04:16:12.859 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:12.858+0000 7fd8c91c4700 1 -- 172.21.15.183:0/4136821576 --> v1:172.21.15.16:6800/4042489921 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fd8b0000bc0 con 0x7fd8ac06e970 2024-03-26T04:16:13.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:12 smithi016 ceph-mon[37215]: pgmap v134: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:16:13.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:12 smithi016 ceph-mon[37215]: Detected new or changed devices on smithi183 2024-03-26T04:16:13.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:12 smithi016 ceph-mon[37215]: Adjusting osd_memory_target on smithi183 to 8544M 2024-03-26T04:16:13.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:12 smithi016 ceph-mon[37215]: osdmap e41: 6 total, 6 up, 6 in 2024-03-26T04:16:13.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:12 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:16:13.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:12 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:16:13.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:12 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:16:13.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:12 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:13.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:12 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:16:13.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:12 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:16:13.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:12 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:16:13.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:12 smithi016 ceph-mon[29101]: pgmap v134: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:16:13.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:12 smithi016 ceph-mon[29101]: Detected new or changed devices on smithi183 2024-03-26T04:16:13.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:12 smithi016 ceph-mon[29101]: Adjusting osd_memory_target on smithi183 to 8544M 2024-03-26T04:16:13.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:12 smithi016 ceph-mon[29101]: osdmap e41: 6 total, 6 up, 6 in 2024-03-26T04:16:13.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:12 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:16:13.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:12 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:16:13.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:12 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:16:13.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:12 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:13.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:12 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:16:13.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:12 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:16:13.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:12 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:16:13.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:12 smithi183 ceph-mon[33745]: pgmap v134: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:16:13.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:12 smithi183 ceph-mon[33745]: Detected new or changed devices on smithi183 2024-03-26T04:16:13.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:12 smithi183 ceph-mon[33745]: Adjusting osd_memory_target on smithi183 to 8544M 2024-03-26T04:16:13.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:12 smithi183 ceph-mon[33745]: osdmap e41: 6 total, 6 up, 6 in 2024-03-26T04:16:13.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:12 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:16:13.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:12 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:16:13.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:12 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:16:13.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:12 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:13.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:12 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:16:13.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:12 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:16:13.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:12 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:16:14.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:13 smithi183 ceph-mon[33745]: from='client.24280 v1:172.21.15.183:0/4136821576' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:16:14.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:13 smithi183 ceph-mon[33745]: osdmap e42: 6 total, 6 up, 6 in 2024-03-26T04:16:14.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:13 smithi016 ceph-mon[29101]: from='client.24280 v1:172.21.15.183:0/4136821576' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:16:14.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:13 smithi016 ceph-mon[29101]: osdmap e42: 6 total, 6 up, 6 in 2024-03-26T04:16:14.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:13 smithi016 ceph-mon[37215]: from='client.24280 v1:172.21.15.183:0/4136821576' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:16:14.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:13 smithi016 ceph-mon[37215]: osdmap e42: 6 total, 6 up, 6 in 2024-03-26T04:16:15.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:14 smithi016 ceph-mon[29101]: pgmap v137: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:16:15.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:14 smithi016 ceph-mon[37215]: pgmap v137: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:16:15.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:14 smithi183 ceph-mon[33745]: pgmap v137: 1 pgs: 1 unknown; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:16:16.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:15 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.183:0/2569511075' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0de5f3f1-7a6e-4d11-9f99-db09de758531"}]: dispatch 2024-03-26T04:16:16.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:15 smithi016 ceph-mon[29101]: from='client.24286 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0de5f3f1-7a6e-4d11-9f99-db09de758531"}]: dispatch 2024-03-26T04:16:16.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:15 smithi016 ceph-mon[29101]: from='client.24286 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0de5f3f1-7a6e-4d11-9f99-db09de758531"}]': finished 2024-03-26T04:16:16.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:15 smithi016 ceph-mon[29101]: osdmap e43: 7 total, 6 up, 7 in 2024-03-26T04:16:16.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:15 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:16:16.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:15 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.183:0/2569511075' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0de5f3f1-7a6e-4d11-9f99-db09de758531"}]: dispatch 2024-03-26T04:16:16.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:15 smithi016 ceph-mon[37215]: from='client.24286 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0de5f3f1-7a6e-4d11-9f99-db09de758531"}]: dispatch 2024-03-26T04:16:16.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:15 smithi016 ceph-mon[37215]: from='client.24286 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0de5f3f1-7a6e-4d11-9f99-db09de758531"}]': finished 2024-03-26T04:16:16.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:15 smithi016 ceph-mon[37215]: osdmap e43: 7 total, 6 up, 7 in 2024-03-26T04:16:16.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:15 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:16:16.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:15 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.183:0/2569511075' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0de5f3f1-7a6e-4d11-9f99-db09de758531"}]: dispatch 2024-03-26T04:16:16.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:15 smithi183 ceph-mon[33745]: from='client.24286 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0de5f3f1-7a6e-4d11-9f99-db09de758531"}]: dispatch 2024-03-26T04:16:16.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:15 smithi183 ceph-mon[33745]: from='client.24286 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0de5f3f1-7a6e-4d11-9f99-db09de758531"}]': finished 2024-03-26T04:16:16.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:15 smithi183 ceph-mon[33745]: osdmap e43: 7 total, 6 up, 7 in 2024-03-26T04:16:16.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:15 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:16:17.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:16 smithi016 ceph-mon[29101]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-03-26T04:16:17.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:16 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.183:0/2653490917' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:16:17.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:16 smithi016 ceph-mon[37215]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-03-26T04:16:17.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:16 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.183:0/2653490917' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:16:17.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:16 smithi183 ceph-mon[33745]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 117 KiB/s, 0 objects/s recovering 2024-03-26T04:16:17.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:16 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.183:0/2653490917' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:16:19.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:18 smithi183 ceph-mon[33745]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T04:16:19.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:18 smithi016 ceph-mon[29101]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T04:16:19.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:18 smithi016 ceph-mon[37215]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T04:16:20.989 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:20 smithi183 ceph-mon[33745]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-26T04:16:21.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:20 smithi016 ceph-mon[29101]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-26T04:16:21.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:20 smithi016 ceph-mon[37215]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-26T04:16:22.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:21 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T04:16:22.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:21 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:16:22.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:21 smithi016 ceph-mon[29101]: Deploying daemon osd.6 on smithi183 2024-03-26T04:16:22.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:21 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T04:16:22.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:21 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:16:22.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:21 smithi016 ceph-mon[37215]: Deploying daemon osd.6 on smithi183 2024-03-26T04:16:22.236 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:21 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-26T04:16:22.236 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:21 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:16:22.236 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:21 smithi183 ceph-mon[33745]: Deploying daemon osd.6 on smithi183 2024-03-26T04:16:23.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:22 smithi016 ceph-mon[29101]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-26T04:16:23.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:22 smithi016 ceph-mon[37215]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-26T04:16:23.239 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:22 smithi183 ceph-mon[33745]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-03-26T04:16:25.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:24 smithi016 ceph-mon[29101]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T04:16:25.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:24 smithi016 ceph-mon[37215]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T04:16:25.233 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:24 smithi183 ceph-mon[33745]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T04:16:27.111 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:26 smithi183 ceph-mon[33745]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:16:27.111 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:26 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:27.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:26 smithi016 ceph-mon[37215]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:16:27.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:26 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:27.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:26 smithi016 ceph-mon[29101]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:16:27.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:26 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:28.223 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:27 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:28.224 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:27 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:16:28.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:27 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:28.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:27 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:16:28.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:27 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:28.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:27 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:16:29.182 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:28 smithi183 ceph-mon[33745]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:16:29.313 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:28 smithi016 ceph-mon[37215]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:16:29.314 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:28 smithi016 ceph-mon[29101]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:16:30.523 INFO:teuthology.orchestra.run.smithi183.stdout:Created osd(s) 6 on host 'smithi183' 2024-03-26T04:16:30.523 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:30.521+0000 7fd8abfff700 1 -- 172.21.15.183:0/4136821576 <== mgr.14152 v1:172.21.15.16:6800/4042489921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 4118428161) 0x7fd8b0000bc0 con 0x7fd8ac06e970 2024-03-26T04:16:30.524 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:30.524+0000 7fd8c91c4700 1 -- 172.21.15.183:0/4136821576 >> v1:172.21.15.16:6800/4042489921 conn(0x7fd8ac06e970 legacy=0x7fd8ac070e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:16:30.525 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:30.524+0000 7fd8c91c4700 1 -- 172.21.15.183:0/4136821576 >> v1:172.21.15.183:6789/0 conn(0x7fd8c4109fd0 legacy=0x7fd8c41a2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:16:30.525 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:30.524+0000 7fd8c91c4700 1 -- 172.21.15.183:0/4136821576 shutdown_connections 2024-03-26T04:16:30.525 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:30.524+0000 7fd8c91c4700 1 -- 172.21.15.183:0/4136821576 >> 172.21.15.183:0/4136821576 conn(0x7fd8c40fbaf0 msgr2=0x7fd8c40fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:16:30.525 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:30.524+0000 7fd8c91c4700 1 -- 172.21.15.183:0/4136821576 shutdown_connections 2024-03-26T04:16:30.525 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:30.524+0000 7fd8c91c4700 1 -- 172.21.15.183:0/4136821576 wait complete. 2024-03-26T04:16:30.757 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:30 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:30.758 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:30 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:30.758 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:30 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:16:30.758 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:30 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:16:30.758 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:30 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:30.758 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:30 smithi016 ceph-mon[37215]: from='osd.6 v1:172.21.15.183:6808/777758685' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T04:16:30.758 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:30 smithi016 ceph-mon[37215]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T04:16:30.758 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:30 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:30.758 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:30 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:30.758 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:30 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:30.759 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:30 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:16:30.759 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:30 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:16:30.759 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:30 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:30.759 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:30 smithi016 ceph-mon[29101]: from='osd.6 v1:172.21.15.183:6808/777758685' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T04:16:30.759 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:30 smithi016 ceph-mon[29101]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T04:16:30.759 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:30 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:30.784 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:30 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:30.784 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:30 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:30.784 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:30 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:16:30.784 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:30 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:16:30.784 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:30 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:30.784 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:30 smithi183 ceph-mon[33745]: from='osd.6 v1:172.21.15.183:6808/777758685' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T04:16:30.784 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:30 smithi183 ceph-mon[33745]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-26T04:16:30.784 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:30 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:31.084 DEBUG:teuthology.orchestra.run.smithi183:osd.6> sudo journalctl -f -n 0 -u ceph-d7d59826-eb26-11ee-95d0-87774f69a715@osd.6.service 2024-03-26T04:16:31.086 INFO:tasks.cephadm:Deploying osd.7 on smithi183 with /dev/vg_nvme/lv_1... 2024-03-26T04:16:31.086 DEBUG:teuthology.orchestra.run.smithi183:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-26T04:16:31.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:31 smithi183 ceph-mon[33745]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:16:31.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:31 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:31.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:31 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:16:31.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:31 smithi183 ceph-mon[33745]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T04:16:31.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:31 smithi183 ceph-mon[33745]: osdmap e44: 7 total, 6 up, 7 in 2024-03-26T04:16:31.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:31 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:16:31.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:31 smithi183 ceph-mon[33745]: from='osd.6 v1:172.21.15.183:6808/777758685' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-26T04:16:31.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:31 smithi183 ceph-mon[33745]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-26T04:16:31.795 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:16:31.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:31 smithi016 ceph-mon[29101]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:16:31.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:31 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:31.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:31 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:16:31.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:31 smithi016 ceph-mon[29101]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T04:16:31.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:31 smithi016 ceph-mon[29101]: osdmap e44: 7 total, 6 up, 7 in 2024-03-26T04:16:31.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:31 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:16:31.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:31 smithi016 ceph-mon[29101]: from='osd.6 v1:172.21.15.183:6808/777758685' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-26T04:16:31.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:31 smithi016 ceph-mon[29101]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-26T04:16:31.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:31 smithi016 ceph-mon[37215]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:16:31.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:31 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:31.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:31 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:16:31.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:31 smithi016 ceph-mon[37215]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-26T04:16:31.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:31 smithi016 ceph-mon[37215]: osdmap e44: 7 total, 6 up, 7 in 2024-03-26T04:16:31.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:31 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:16:31.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:31 smithi016 ceph-mon[37215]: from='osd.6 v1:172.21.15.183:6808/777758685' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-26T04:16:31.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:31 smithi016 ceph-mon[37215]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-26T04:16:32.233 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:16:31 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:16:31.965+0000 7fcb6b1da700 -1 osd.6 0 waiting for initial osdmap 2024-03-26T04:16:32.233 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:16:31 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:16:31.971+0000 7fcb62fc5700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T04:16:32.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:32 smithi183 ceph-mon[33745]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-03-26T04:16:32.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:32 smithi183 ceph-mon[33745]: osdmap e45: 7 total, 6 up, 7 in 2024-03-26T04:16:32.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:32 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:16:32.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:32 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:16:32.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:32 smithi183 ceph-mon[33745]: from='osd.6 ' entity='osd.6' 2024-03-26T04:16:32.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:32 smithi016 ceph-mon[37215]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-03-26T04:16:32.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:32 smithi016 ceph-mon[37215]: osdmap e45: 7 total, 6 up, 7 in 2024-03-26T04:16:32.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:32 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:16:32.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:32 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:16:32.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:32 smithi016 ceph-mon[37215]: from='osd.6 ' entity='osd.6' 2024-03-26T04:16:32.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:32 smithi016 ceph-mon[29101]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-03-26T04:16:32.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:32 smithi016 ceph-mon[29101]: osdmap e45: 7 total, 6 up, 7 in 2024-03-26T04:16:32.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:32 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:16:32.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:32 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:16:32.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:32 smithi016 ceph-mon[29101]: from='osd.6 ' entity='osd.6' 2024-03-26T04:16:33.721 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:33 smithi183 ceph-mon[33745]: purged_snaps scrub starts 2024-03-26T04:16:33.721 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:33 smithi183 ceph-mon[33745]: purged_snaps scrub ok 2024-03-26T04:16:33.721 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:33 smithi183 ceph-mon[33745]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:16:33.721 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:33 smithi183 ceph-mon[33745]: osd.6 v1:172.21.15.183:6808/777758685 boot 2024-03-26T04:16:33.721 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:33 smithi183 ceph-mon[33745]: osdmap e46: 7 total, 7 up, 7 in 2024-03-26T04:16:33.721 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:33 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:16:33.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:33 smithi016 ceph-mon[37215]: purged_snaps scrub starts 2024-03-26T04:16:33.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:33 smithi016 ceph-mon[37215]: purged_snaps scrub ok 2024-03-26T04:16:33.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:33 smithi016 ceph-mon[37215]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:16:33.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:33 smithi016 ceph-mon[37215]: osd.6 v1:172.21.15.183:6808/777758685 boot 2024-03-26T04:16:33.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:33 smithi016 ceph-mon[37215]: osdmap e46: 7 total, 7 up, 7 in 2024-03-26T04:16:33.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:33 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:16:33.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:33 smithi016 ceph-mon[29101]: purged_snaps scrub starts 2024-03-26T04:16:33.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:33 smithi016 ceph-mon[29101]: purged_snaps scrub ok 2024-03-26T04:16:33.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:33 smithi016 ceph-mon[29101]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 170 MiB used, 536 GiB / 536 GiB avail 2024-03-26T04:16:33.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:33 smithi016 ceph-mon[29101]: osd.6 v1:172.21.15.183:6808/777758685 boot 2024-03-26T04:16:33.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:33 smithi016 ceph-mon[29101]: osdmap e46: 7 total, 7 up, 7 in 2024-03-26T04:16:33.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:33 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:16:34.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:34 smithi183 ceph-mon[33745]: osdmap e47: 7 total, 7 up, 7 in 2024-03-26T04:16:34.792 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-26T04:16:34.792 INFO:teuthology.orchestra.run.smithi183.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-26T04:16:34.792 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10+0 records in 2024-03-26T04:16:34.792 INFO:teuthology.orchestra.run.smithi183.stderr:10+0 records out 2024-03-26T04:16:34.793 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0120425 s, 871 MB/s 2024-03-26T04:16:34.793 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping successful for: 2024-03-26T04:16:34.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:34 smithi016 ceph-mon[29101]: osdmap e47: 7 total, 7 up, 7 in 2024-03-26T04:16:34.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:34 smithi016 ceph-mon[37215]: osdmap e47: 7 total, 7 up, 7 in 2024-03-26T04:16:35.416 DEBUG:teuthology.orchestra.run.smithi183:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph orch daemon add osd smithi183:vg_nvme/lv_1 2024-03-26T04:16:35.653 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:16:35.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:35 smithi183 ceph-mon[33745]: pgmap v152: 1 pgs: 1 remapped+peering; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:16:35.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:35 smithi183 ceph-mon[33745]: osdmap e48: 7 total, 7 up, 7 in 2024-03-26T04:16:35.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:35 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:35.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:35 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:35.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:35 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:16:35.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:35 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:16:35.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:35 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:16:35.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:35 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:35.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:35 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:16:35.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:35 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:16:35.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:35 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:35.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:35 smithi183 ceph-mon[33745]: osdmap e49: 7 total, 7 up, 7 in 2024-03-26T04:16:35.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:35 smithi016 ceph-mon[29101]: pgmap v152: 1 pgs: 1 remapped+peering; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:16:35.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:35 smithi016 ceph-mon[29101]: osdmap e48: 7 total, 7 up, 7 in 2024-03-26T04:16:35.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:35 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:35.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:35 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:35.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:35 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:16:35.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:35 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:16:35.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:35 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:16:35.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:35 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:35.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:35 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:16:35.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:35 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:16:35.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:35 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:35.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:35 smithi016 ceph-mon[29101]: osdmap e49: 7 total, 7 up, 7 in 2024-03-26T04:16:35.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:35 smithi016 ceph-mon[37215]: pgmap v152: 1 pgs: 1 remapped+peering; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:16:35.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:35 smithi016 ceph-mon[37215]: osdmap e48: 7 total, 7 up, 7 in 2024-03-26T04:16:35.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:35 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:35.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:35 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:35.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:35 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:16:35.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:35 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:16:35.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:35 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:16:35.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:35 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:35.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:35 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:16:35.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:35 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:16:35.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:35 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:35.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:35 smithi016 ceph-mon[37215]: osdmap e49: 7 total, 7 up, 7 in 2024-03-26T04:16:36.629 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:36 smithi183 ceph-mon[33745]: Detected new or changed devices on smithi183 2024-03-26T04:16:36.629 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:36 smithi183 ceph-mon[33745]: Adjusting osd_memory_target on smithi183 to 5696M 2024-03-26T04:16:36.630 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:36 smithi183 ceph-mon[33745]: osdmap e50: 7 total, 7 up, 7 in 2024-03-26T04:16:36.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:36 smithi016 ceph-mon[29101]: Detected new or changed devices on smithi183 2024-03-26T04:16:36.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:36 smithi016 ceph-mon[29101]: Adjusting osd_memory_target on smithi183 to 5696M 2024-03-26T04:16:36.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:36 smithi016 ceph-mon[29101]: osdmap e50: 7 total, 7 up, 7 in 2024-03-26T04:16:36.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:36 smithi016 ceph-mon[37215]: Detected new or changed devices on smithi183 2024-03-26T04:16:36.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:36 smithi016 ceph-mon[37215]: Adjusting osd_memory_target on smithi183 to 5696M 2024-03-26T04:16:36.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:36 smithi016 ceph-mon[37215]: osdmap e50: 7 total, 7 up, 7 in 2024-03-26T04:16:37.150 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:37.149+0000 7f9a28c9e700 1 -- 172.21.15.183:0/4032421153 >> v1:172.21.15.183:6789/0 conn(0x7f9a241051a0 legacy=0x7f9a241075f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:16:37.150 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:37.149+0000 7f9a28c9e700 1 -- 172.21.15.183:0/4032421153 shutdown_connections 2024-03-26T04:16:37.150 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:37.149+0000 7f9a28c9e700 1 -- 172.21.15.183:0/4032421153 >> 172.21.15.183:0/4032421153 conn(0x7f9a240fc310 msgr2=0x7f9a240fe750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:16:37.150 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:37.149+0000 7f9a28c9e700 1 -- 172.21.15.183:0/4032421153 shutdown_connections 2024-03-26T04:16:37.150 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:37.149+0000 7f9a28c9e700 1 -- 172.21.15.183:0/4032421153 wait complete. 2024-03-26T04:16:37.153 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:37.150+0000 7f9a28c9e700 1 Processor -- start 2024-03-26T04:16:37.153 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:37.150+0000 7f9a28c9e700 1 -- start start 2024-03-26T04:16:37.153 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:37.150+0000 7f9a28c9e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9a2406a920 con 0x7f9a24100d70 2024-03-26T04:16:37.153 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:37.150+0000 7f9a28c9e700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9a241a34c0 con 0x7f9a241051a0 2024-03-26T04:16:37.153 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:37.150+0000 7f9a28c9e700 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f9a241a46c0 con 0x7f9a24108ee0 2024-03-26T04:16:37.153 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:37.150+0000 7f9a21d9b700 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f9a241051a0 0x7f9a2419d660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:49926/0 (socket says 172.21.15.183:49926) 2024-03-26T04:16:37.153 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:37.150+0000 7f9a21d9b700 1 -- 172.21.15.183:0/3256038529 learned_addr learned my addr 172.21.15.183:0/3256038529 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-26T04:16:37.153 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:37.151+0000 7f9a137fe700 1 -- 172.21.15.183:0/3256038529 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1751636659 0 0) 0x7f9a241a34c0 con 0x7f9a241051a0 2024-03-26T04:16:37.154 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:37.151+0000 7f9a137fe700 1 -- 172.21.15.183:0/3256038529 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a0c0035b0 con 0x7f9a241051a0 2024-03-26T04:16:37.154 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:37.151+0000 7f9a137fe700 1 -- 172.21.15.183:0/3256038529 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 976149186 0 0) 0x7f9a241a46c0 con 0x7f9a24108ee0 2024-03-26T04:16:37.154 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:37.151+0000 7f9a137fe700 1 -- 172.21.15.183:0/3256038529 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a241a34c0 con 0x7f9a24108ee0 2024-03-26T04:16:37.154 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:37.151+0000 7f9a137fe700 1 -- 172.21.15.183:0/3256038529 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3667400453 0 0) 0x7f9a2406a920 con 0x7f9a24100d70 2024-03-26T04:16:37.154 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:37.151+0000 7f9a137fe700 1 -- 172.21.15.183:0/3256038529 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a241a46c0 con 0x7f9a24100d70 2024-03-26T04:16:37.154 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:37.151+0000 7f9a137fe700 1 -- 172.21.15.183:0/3256038529 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 555721499 0 0) 0x7f9a0c0035b0 con 0x7f9a241051a0 2024-03-26T04:16:37.154 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:37.151+0000 7f9a137fe700 1 -- 172.21.15.183:0/3256038529 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a2406a920 con 0x7f9a241051a0 2024-03-26T04:16:37.154 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:37.151+0000 7f9a137fe700 1 -- 172.21.15.183:0/3256038529 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f9a18001aa0 con 0x7f9a241051a0 2024-03-26T04:16:37.154 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:37.151+0000 7f9a137fe700 1 -- 172.21.15.183:0/3256038529 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1838572186 0 0) 0x7f9a241a34c0 con 0x7f9a24108ee0 2024-03-26T04:16:37.154 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:37.151+0000 7f9a137fe700 1 -- 172.21.15.183:0/3256038529 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a0c0035b0 con 0x7f9a24108ee0 2024-03-26T04:16:37.155 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:37.151+0000 7f9a137fe700 1 -- 172.21.15.183:0/3256038529 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f9a1c001aa0 con 0x7f9a24108ee0 2024-03-26T04:16:37.155 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:37.151+0000 7f9a137fe700 1 -- 172.21.15.183:0/3256038529 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 374697558 0 0) 0x7f9a2406a920 con 0x7f9a241051a0 2024-03-26T04:16:37.155 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:37.151+0000 7f9a137fe700 1 -- 172.21.15.183:0/3256038529 >> v1:172.21.15.16:6790/0 conn(0x7f9a24108ee0 legacy=0x7f9a241a0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:16:37.155 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:37.151+0000 7f9a137fe700 1 -- 172.21.15.183:0/3256038529 >> v1:172.21.15.16:6789/0 conn(0x7f9a24100d70 legacy=0x7f9a24199f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:16:37.155 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:37.151+0000 7f9a137fe700 1 -- 172.21.15.183:0/3256038529 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a241a58c0 con 0x7f9a241051a0 2024-03-26T04:16:37.156 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:37.151+0000 7f9a28c9e700 1 -- 172.21.15.183:0/3256038529 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a241a48b0 con 0x7f9a241051a0 2024-03-26T04:16:37.156 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:37.151+0000 7f9a28c9e700 1 -- 172.21.15.183:0/3256038529 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a241a4db0 con 0x7f9a241051a0 2024-03-26T04:16:37.156 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:37.151+0000 7f9a137fe700 1 -- 172.21.15.183:0/3256038529 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f9a18000fe0 con 0x7f9a241051a0 2024-03-26T04:16:37.156 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:37.151+0000 7f9a137fe700 1 -- 172.21.15.183:0/3256038529 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f9a18004f30 con 0x7f9a241051a0 2024-03-26T04:16:37.156 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:37.152+0000 7f9a28c9e700 1 -- 172.21.15.183:0/3256038529 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a04005150 con 0x7f9a241051a0 2024-03-26T04:16:37.156 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:37.153+0000 7f9a137fe700 1 -- 172.21.15.183:0/3256038529 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 15) v1 ==== 91363+0+0 (unknown 160314185 0 0) 0x7f9a18004710 con 0x7f9a241051a0 2024-03-26T04:16:37.156 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:37.153+0000 7f9a137fe700 1 -- 172.21.15.183:0/3256038529 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(50..50 src has 1..50) v4 ==== 3789+0+0 (unknown 4196200710 0 0) 0x7f9a18088260 con 0x7f9a241051a0 2024-03-26T04:16:37.158 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:37.158+0000 7f9a137fe700 1 -- 172.21.15.183:0/3256038529 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f9a18052350 con 0x7f9a241051a0 2024-03-26T04:16:37.361 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:37.360+0000 7f9a28c9e700 1 -- 172.21.15.183:0/3256038529 --> v1:172.21.15.16:6800/4042489921 -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f9a04000bc0 con 0x7f9a0c06e710 2024-03-26T04:16:37.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:37 smithi183 ceph-mon[33745]: pgmap v155: 1 pgs: 1 remapped+peering; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:16:37.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:37 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:16:37.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:37 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:16:37.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:37 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:16:37.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:37 smithi016 ceph-mon[29101]: pgmap v155: 1 pgs: 1 remapped+peering; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:16:37.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:37 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:16:37.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:37 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:16:37.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:37 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:16:37.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:37 smithi016 ceph-mon[37215]: pgmap v155: 1 pgs: 1 remapped+peering; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:16:37.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:37 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-26T04:16:37.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:37 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-26T04:16:37.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:37 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:16:38.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:38 smithi016 ceph-mon[29101]: from='client.24307 v1:172.21.15.183:0/3256038529' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:16:38.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:38 smithi016 ceph-mon[37215]: from='client.24307 v1:172.21.15.183:0/3256038529' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:16:38.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:38 smithi183 ceph-mon[33745]: from='client.24307 v1:172.21.15.183:0/3256038529' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:16:39.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:39 smithi016 ceph-mon[29101]: pgmap v157: 1 pgs: 1 remapped+peering; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:16:39.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:39 smithi016 ceph-mon[37215]: pgmap v157: 1 pgs: 1 remapped+peering; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:16:40.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:39 smithi183 ceph-mon[33745]: pgmap v157: 1 pgs: 1 remapped+peering; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:16:40.829 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:40 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.183:0/2059218210' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6091e9e5-f241-4eed-81dd-970138c422c0"}]: dispatch 2024-03-26T04:16:40.829 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:40 smithi016 ceph-mon[37215]: from='client.24313 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6091e9e5-f241-4eed-81dd-970138c422c0"}]: dispatch 2024-03-26T04:16:40.829 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:40 smithi016 ceph-mon[37215]: from='client.24313 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6091e9e5-f241-4eed-81dd-970138c422c0"}]': finished 2024-03-26T04:16:40.829 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:40 smithi016 ceph-mon[37215]: osdmap e51: 8 total, 7 up, 8 in 2024-03-26T04:16:40.829 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:40 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:16:40.829 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:40 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.183:0/2059218210' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6091e9e5-f241-4eed-81dd-970138c422c0"}]: dispatch 2024-03-26T04:16:40.829 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:40 smithi016 ceph-mon[29101]: from='client.24313 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6091e9e5-f241-4eed-81dd-970138c422c0"}]: dispatch 2024-03-26T04:16:40.830 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:40 smithi016 ceph-mon[29101]: from='client.24313 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6091e9e5-f241-4eed-81dd-970138c422c0"}]': finished 2024-03-26T04:16:40.830 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:40 smithi016 ceph-mon[29101]: osdmap e51: 8 total, 7 up, 8 in 2024-03-26T04:16:40.830 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:40 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:16:41.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:40 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.183:0/2059218210' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6091e9e5-f241-4eed-81dd-970138c422c0"}]: dispatch 2024-03-26T04:16:41.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:40 smithi183 ceph-mon[33745]: from='client.24313 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6091e9e5-f241-4eed-81dd-970138c422c0"}]: dispatch 2024-03-26T04:16:41.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:40 smithi183 ceph-mon[33745]: from='client.24313 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6091e9e5-f241-4eed-81dd-970138c422c0"}]': finished 2024-03-26T04:16:41.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:40 smithi183 ceph-mon[33745]: osdmap e51: 8 total, 7 up, 8 in 2024-03-26T04:16:41.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:40 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:16:41.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:41 smithi016 ceph-mon[29101]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T04:16:41.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:41 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.183:0/2370894230' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:16:41.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:41 smithi016 ceph-mon[37215]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T04:16:41.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:41 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.183:0/2370894230' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:16:42.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:41 smithi183 ceph-mon[33745]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 198 MiB used, 626 GiB / 626 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T04:16:42.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:41 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.183:0/2370894230' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-26T04:16:43.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:43 smithi016 ceph-mon[29101]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-03-26T04:16:43.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:43 smithi016 ceph-mon[37215]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-03-26T04:16:44.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:43 smithi183 ceph-mon[33745]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-03-26T04:16:45.784 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:45 smithi183 ceph-mon[33745]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-26T04:16:45.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:45 smithi016 ceph-mon[29101]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-26T04:16:45.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:45 smithi016 ceph-mon[37215]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-26T04:16:46.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:46 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T04:16:46.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:46 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:16:46.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:46 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T04:16:46.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:46 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:16:46.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:46 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-26T04:16:46.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:46 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:16:47.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:47 smithi183 ceph-mon[33745]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-26T04:16:47.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:47 smithi183 ceph-mon[33745]: Deploying daemon osd.7 on smithi183 2024-03-26T04:16:47.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:47 smithi016 ceph-mon[29101]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-26T04:16:47.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:47 smithi016 ceph-mon[29101]: Deploying daemon osd.7 on smithi183 2024-03-26T04:16:47.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:47 smithi016 ceph-mon[37215]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-26T04:16:47.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:47 smithi016 ceph-mon[37215]: Deploying daemon osd.7 on smithi183 2024-03-26T04:16:49.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:49 smithi183 ceph-mon[33745]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T04:16:49.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:49 smithi016 ceph-mon[29101]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T04:16:49.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:49 smithi016 ceph-mon[37215]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-26T04:16:51.692 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:51 smithi183 ceph-mon[33745]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:16:51.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:51 smithi016 ceph-mon[29101]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:16:51.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:51 smithi016 ceph-mon[37215]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:16:53.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:52 smithi183 ceph-mon[33745]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:16:53.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:52 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:53.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:52 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:53.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:52 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:16:53.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:52 smithi016 ceph-mon[29101]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:16:53.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:52 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:53.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:52 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:53.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:52 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:16:53.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:52 smithi016 ceph-mon[37215]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:16:53.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:52 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:53.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:52 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:53.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:52 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:16:54.921 INFO:teuthology.orchestra.run.smithi183.stdout:Created osd(s) 7 on host 'smithi183' 2024-03-26T04:16:54.921 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:54.920+0000 7f9a137fe700 1 -- 172.21.15.183:0/3256038529 <== mgr.14152 v1:172.21.15.16:6800/4042489921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (unknown 0 0 2842678453) 0x7f9a04000bc0 con 0x7f9a0c06e710 2024-03-26T04:16:54.923 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:54.922+0000 7f9a28c9e700 1 -- 172.21.15.183:0/3256038529 >> v1:172.21.15.16:6800/4042489921 conn(0x7f9a0c06e710 legacy=0x7f9a0c070bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:16:54.923 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:54.922+0000 7f9a28c9e700 1 -- 172.21.15.183:0/3256038529 >> v1:172.21.15.183:6789/0 conn(0x7f9a241051a0 legacy=0x7f9a2419d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:16:54.923 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:54.923+0000 7f9a28c9e700 1 -- 172.21.15.183:0/3256038529 shutdown_connections 2024-03-26T04:16:54.923 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:54.923+0000 7f9a28c9e700 1 -- 172.21.15.183:0/3256038529 >> 172.21.15.183:0/3256038529 conn(0x7f9a240fc310 msgr2=0x7f9a241075f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:16:54.923 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:54.923+0000 7f9a28c9e700 1 -- 172.21.15.183:0/3256038529 shutdown_connections 2024-03-26T04:16:54.924 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:16:54.923+0000 7f9a28c9e700 1 -- 172.21.15.183:0/3256038529 wait complete. 2024-03-26T04:16:55.264 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:54 smithi183 ceph-mon[33745]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:16:55.265 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:54 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:55.265 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:54 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:55.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:54 smithi016 ceph-mon[29101]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:16:55.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:54 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:55.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:54 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:55.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:54 smithi016 ceph-mon[37215]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:16:55.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:54 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:55.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:54 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:55.976 DEBUG:teuthology.orchestra.run.smithi183:osd.7> sudo journalctl -f -n 0 -u ceph-d7d59826-eb26-11ee-95d0-87774f69a715@osd.7.service 2024-03-26T04:16:55.978 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-26T04:16:55.979 DEBUG:teuthology.orchestra.run.smithi016:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph osd stat -f json 2024-03-26T04:16:56.240 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:16:56.273 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:55 smithi183 ceph-mon[33745]: from='osd.7 v1:172.21.15.183:6812/804876304' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T04:16:56.274 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:55 smithi183 ceph-mon[33745]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T04:16:56.362 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:55 smithi016 ceph-mon[29101]: from='osd.7 v1:172.21.15.183:6812/804876304' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T04:16:56.363 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:55 smithi016 ceph-mon[29101]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T04:16:56.363 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:55 smithi016 ceph-mon[37215]: from='osd.7 v1:172.21.15.183:6812/804876304' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T04:16:56.363 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:55 smithi016 ceph-mon[37215]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-26T04:16:57.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:56 smithi183 ceph-mon[33745]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:16:57.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:56 smithi183 ceph-mon[33745]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T04:16:57.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:56 smithi183 ceph-mon[33745]: from='osd.7 v1:172.21.15.183:6812/804876304' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-26T04:16:57.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:56 smithi183 ceph-mon[33745]: osdmap e52: 8 total, 7 up, 8 in 2024-03-26T04:16:57.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:56 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:16:57.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:56 smithi183 ceph-mon[33745]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-26T04:16:57.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:56 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:57.238 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:56 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:57.333 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:56 smithi016 ceph-mon[37215]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:16:57.333 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:56 smithi016 ceph-mon[37215]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T04:16:57.333 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:56 smithi016 ceph-mon[37215]: from='osd.7 v1:172.21.15.183:6812/804876304' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-26T04:16:57.333 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:56 smithi016 ceph-mon[37215]: osdmap e52: 8 total, 7 up, 8 in 2024-03-26T04:16:57.333 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:56 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:16:57.333 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:56 smithi016 ceph-mon[37215]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-26T04:16:57.333 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:56 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:57.333 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:56 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:57.333 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:56 smithi016 ceph-mon[29101]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:16:57.334 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:56 smithi016 ceph-mon[29101]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-26T04:16:57.334 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:56 smithi016 ceph-mon[29101]: from='osd.7 v1:172.21.15.183:6812/804876304' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-26T04:16:57.334 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:56 smithi016 ceph-mon[29101]: osdmap e52: 8 total, 7 up, 8 in 2024-03-26T04:16:57.334 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:56 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:16:57.334 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:56 smithi016 ceph-mon[29101]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-03-26T04:16:57.334 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:56 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:57.334 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:56 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:16:57.511 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:16:57 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:16:57.385+0000 7f37f9906700 -1 osd.7 0 waiting for initial osdmap 2024-03-26T04:16:57.511 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:16:57 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:16:57.391+0000 7f37f16f1700 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-26T04:16:57.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:57.829+0000 7fc7f6a0a700 1 -- 172.21.15.16:0/3745200273 >> v1:172.21.15.16:6789/0 conn(0x7fc7f0100190 legacy=0x7fc7f01005b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:16:57.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:57.829+0000 7fc7f6a0a700 1 -- 172.21.15.16:0/3745200273 shutdown_connections 2024-03-26T04:16:57.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:57.829+0000 7fc7f6a0a700 1 -- 172.21.15.16:0/3745200273 >> 172.21.15.16:0/3745200273 conn(0x7fc7f00fbaf0 msgr2=0x7fc7f00fdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:16:57.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:57.829+0000 7fc7f6a0a700 1 -- 172.21.15.16:0/3745200273 shutdown_connections 2024-03-26T04:16:57.831 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:57.829+0000 7fc7f6a0a700 1 -- 172.21.15.16:0/3745200273 wait complete. 2024-03-26T04:16:57.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:57.830+0000 7fc7f6a0a700 1 Processor -- start 2024-03-26T04:16:57.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:57.830+0000 7fc7f6a0a700 1 -- start start 2024-03-26T04:16:57.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:57.830+0000 7fc7f6a0a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc7f0197c00 con 0x7fc7f0106440 2024-03-26T04:16:57.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:57.830+0000 7fc7f6a0a700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc7f01a33f0 con 0x7fc7f0109fe0 2024-03-26T04:16:57.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:57.830+0000 7fc7f6a0a700 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc7f01a4590 con 0x7fc7f0100190 2024-03-26T04:16:57.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:57.830+0000 7fc7effff700 1 --1- >> v1:172.21.15.16:6790/0 conn(0x7fc7f0100190 0x7fc7f01970b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6790/0 says I am v1:172.21.15.16:50666/0 (socket says 172.21.15.16:50666) 2024-03-26T04:16:57.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:57.830+0000 7fc7effff700 1 -- 172.21.15.16:0/3074035808 learned_addr learned my addr 172.21.15.16:0/3074035808 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:16:57.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:57.831+0000 7fc7ed7fa700 1 -- 172.21.15.16:0/3074035808 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 103301342 0 0) 0x7fc7f01a4590 con 0x7fc7f0100190 2024-03-26T04:16:57.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:57.831+0000 7fc7ed7fa700 1 -- 172.21.15.16:0/3074035808 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7d80035b0 con 0x7fc7f0100190 2024-03-26T04:16:57.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:57.831+0000 7fc7ed7fa700 1 -- 172.21.15.16:0/3074035808 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 836294805 0 0) 0x7fc7f0197c00 con 0x7fc7f0106440 2024-03-26T04:16:57.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:57.831+0000 7fc7ed7fa700 1 -- 172.21.15.16:0/3074035808 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7f01a4590 con 0x7fc7f0106440 2024-03-26T04:16:57.833 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:57.831+0000 7fc7ed7fa700 1 -- 172.21.15.16:0/3074035808 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2855596790 0 0) 0x7fc7f01a33f0 con 0x7fc7f0109fe0 2024-03-26T04:16:57.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:57.831+0000 7fc7ed7fa700 1 -- 172.21.15.16:0/3074035808 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7f0197c00 con 0x7fc7f0109fe0 2024-03-26T04:16:57.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:57.831+0000 7fc7ed7fa700 1 -- 172.21.15.16:0/3074035808 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3151445217 0 0) 0x7fc7d80035b0 con 0x7fc7f0100190 2024-03-26T04:16:57.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:57.831+0000 7fc7ed7fa700 1 -- 172.21.15.16:0/3074035808 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7f01a33f0 con 0x7fc7f0100190 2024-03-26T04:16:57.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:57.831+0000 7fc7ed7fa700 1 -- 172.21.15.16:0/3074035808 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fc7e0004540 con 0x7fc7f0100190 2024-03-26T04:16:57.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:57.831+0000 7fc7ed7fa700 1 -- 172.21.15.16:0/3074035808 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2234383579 0 0) 0x7fc7f01a4590 con 0x7fc7f0106440 2024-03-26T04:16:57.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:57.831+0000 7fc7ed7fa700 1 -- 172.21.15.16:0/3074035808 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7d80035b0 con 0x7fc7f0106440 2024-03-26T04:16:57.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:57.831+0000 7fc7ed7fa700 1 -- 172.21.15.16:0/3074035808 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fc7e40017f0 con 0x7fc7f0106440 2024-03-26T04:16:57.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:57.831+0000 7fc7ed7fa700 1 -- 172.21.15.16:0/3074035808 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3697202374 0 0) 0x7fc7f0197c00 con 0x7fc7f0109fe0 2024-03-26T04:16:57.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:57.831+0000 7fc7ed7fa700 1 -- 172.21.15.16:0/3074035808 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7f01a4590 con 0x7fc7f0109fe0 2024-03-26T04:16:57.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:57.831+0000 7fc7ed7fa700 1 -- 172.21.15.16:0/3074035808 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fc7e8001c00 con 0x7fc7f0109fe0 2024-03-26T04:16:57.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:57.831+0000 7fc7ed7fa700 1 -- 172.21.15.16:0/3074035808 <== mon.2 v1:172.21.15.16:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2449757793 0 0) 0x7fc7f01a33f0 con 0x7fc7f0100190 2024-03-26T04:16:57.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:57.831+0000 7fc7ed7fa700 1 -- 172.21.15.16:0/3074035808 >> v1:172.21.15.183:6789/0 conn(0x7fc7f0109fe0 legacy=0x7fc7f01a0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:16:57.834 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:57.831+0000 7fc7ed7fa700 1 -- 172.21.15.16:0/3074035808 >> v1:172.21.15.16:6789/0 conn(0x7fc7f0106440 legacy=0x7fc7f019d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:16:57.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:57.831+0000 7fc7ed7fa700 1 -- 172.21.15.16:0/3074035808 --> v1:172.21.15.16:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f01a5730 con 0x7fc7f0100190 2024-03-26T04:16:57.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:57.831+0000 7fc7f6a0a700 1 -- 172.21.15.16:0/3074035808 --> v1:172.21.15.16:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7f01a4780 con 0x7fc7f0100190 2024-03-26T04:16:57.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:57.831+0000 7fc7f6a0a700 1 -- 172.21.15.16:0/3074035808 --> v1:172.21.15.16:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7f01a4c80 con 0x7fc7f0100190 2024-03-26T04:16:57.835 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:57.832+0000 7fc7ed7fa700 1 -- 172.21.15.16:0/3074035808 <== mon.2 v1:172.21.15.16:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7fc7e00011e0 con 0x7fc7f0100190 2024-03-26T04:16:57.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:57.832+0000 7fc7ed7fa700 1 -- 172.21.15.16:0/3074035808 <== mon.2 v1:172.21.15.16:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fc7e0005000 con 0x7fc7f0100190 2024-03-26T04:16:57.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:57.833+0000 7fc7f6a0a700 1 -- 172.21.15.16:0/3074035808 --> v1:172.21.15.16:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7dc005150 con 0x7fc7f0100190 2024-03-26T04:16:57.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:57.833+0000 7fc7ed7fa700 1 -- 172.21.15.16:0/3074035808 <== mon.2 v1:172.21.15.16:6790/0 7 ==== mgrmap(e 15) v1 ==== 91363+0+0 (unknown 160314185 0 0) 0x7fc7e0005280 con 0x7fc7f0100190 2024-03-26T04:16:57.836 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:57.834+0000 7fc7ed7fa700 1 -- 172.21.15.16:0/3074035808 <== mon.2 v1:172.21.15.16:6790/0 8 ==== osd_map(53..53 src has 1..53) v4 ==== 3941+0+0 (unknown 1359465056 0 0) 0x7fc7e0088230 con 0x7fc7f0100190 2024-03-26T04:16:57.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:57.839+0000 7fc7ed7fa700 1 -- 172.21.15.16:0/3074035808 <== mon.2 v1:172.21.15.16:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fc7e0053160 con 0x7fc7f0100190 2024-03-26T04:16:58.052 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:58.051+0000 7fc7f6a0a700 1 -- 172.21.15.16:0/3074035808 --> v1:172.21.15.16:6790/0 -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fc7dc005400 con 0x7fc7f0100190 2024-03-26T04:16:58.053 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:58.051+0000 7fc7ed7fa700 1 -- 172.21.15.16:0/3074035808 <== mon.2 v1:172.21.15.16:6790/0 10 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v54) v1 ==== 74+0+130 (unknown 4107983022 0 1342739516) 0x7fc7e0055090 con 0x7fc7f0100190 2024-03-26T04:16:58.053 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:16:58.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:58.053+0000 7fc7f6a0a700 1 -- 172.21.15.16:0/3074035808 >> v1:172.21.15.16:6800/4042489921 conn(0x7fc7d806eab0 legacy=0x7fc7d8070f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:16:58.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:58.053+0000 7fc7f6a0a700 1 -- 172.21.15.16:0/3074035808 >> v1:172.21.15.16:6790/0 conn(0x7fc7f0100190 legacy=0x7fc7f01970b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:16:58.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:58.054+0000 7fc7f6a0a700 1 -- 172.21.15.16:0/3074035808 shutdown_connections 2024-03-26T04:16:58.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:58.054+0000 7fc7f6a0a700 1 -- 172.21.15.16:0/3074035808 >> 172.21.15.16:0/3074035808 conn(0x7fc7f00fbaf0 msgr2=0x7fc7f00fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:16:58.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:58.054+0000 7fc7f6a0a700 1 -- 172.21.15.16:0/3074035808 shutdown_connections 2024-03-26T04:16:58.055 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:16:58.054+0000 7fc7f6a0a700 1 -- 172.21.15.16:0/3074035808 wait complete. 2024-03-26T04:16:58.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:57 smithi183 ceph-mon[33745]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-03-26T04:16:58.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:57 smithi183 ceph-mon[33745]: osdmap e53: 8 total, 7 up, 8 in 2024-03-26T04:16:58.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:57 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:16:58.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:57 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:16:58.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:57 smithi183 ceph-mon[33745]: from='osd.7 ' entity='osd.7' 2024-03-26T04:16:58.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:57 smithi183 ceph-mon[33745]: osd.7 v1:172.21.15.183:6812/804876304 boot 2024-03-26T04:16:58.348 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:57 smithi016 ceph-mon[37215]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-03-26T04:16:58.348 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:57 smithi016 ceph-mon[37215]: osdmap e53: 8 total, 7 up, 8 in 2024-03-26T04:16:58.348 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:57 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:16:58.348 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:57 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:16:58.349 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:57 smithi016 ceph-mon[37215]: from='osd.7 ' entity='osd.7' 2024-03-26T04:16:58.349 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:57 smithi016 ceph-mon[37215]: osd.7 v1:172.21.15.183:6812/804876304 boot 2024-03-26T04:16:58.349 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:57 smithi016 ceph-mon[29101]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-03-26T04:16:58.349 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:57 smithi016 ceph-mon[29101]: osdmap e53: 8 total, 7 up, 8 in 2024-03-26T04:16:58.349 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:57 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:16:58.349 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:57 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:16:58.349 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:57 smithi016 ceph-mon[29101]: from='osd.7 ' entity='osd.7' 2024-03-26T04:16:58.350 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:57 smithi016 ceph-mon[29101]: osd.7 v1:172.21.15.183:6812/804876304 boot 2024-03-26T04:16:58.661 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":54,"num_osds":8,"num_up_osds":8,"osd_up_since":1711426617,"num_in_osds":8,"osd_in_since":1711426600,"num_remapped_pgs":1} 2024-03-26T04:16:58.661 DEBUG:teuthology.orchestra.run.smithi016:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T04:16:58.913 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:16:59.255 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:58 smithi016 ceph-mon[37215]: purged_snaps scrub starts 2024-03-26T04:16:59.255 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:58 smithi016 ceph-mon[37215]: purged_snaps scrub ok 2024-03-26T04:16:59.255 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:58 smithi016 ceph-mon[37215]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:16:59.255 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:58 smithi016 ceph-mon[37215]: osdmap e54: 8 total, 8 up, 8 in 2024-03-26T04:16:59.255 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:58 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:16:59.255 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3074035808' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T04:16:59.256 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:58 smithi016 ceph-mon[29101]: purged_snaps scrub starts 2024-03-26T04:16:59.256 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:58 smithi016 ceph-mon[29101]: purged_snaps scrub ok 2024-03-26T04:16:59.256 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:58 smithi016 ceph-mon[29101]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:16:59.256 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:58 smithi016 ceph-mon[29101]: osdmap e54: 8 total, 8 up, 8 in 2024-03-26T04:16:59.256 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:58 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:16:59.256 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3074035808' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T04:16:59.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:58 smithi183 ceph-mon[33745]: purged_snaps scrub starts 2024-03-26T04:16:59.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:58 smithi183 ceph-mon[33745]: purged_snaps scrub ok 2024-03-26T04:16:59.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:58 smithi183 ceph-mon[33745]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 197 MiB used, 626 GiB / 626 GiB avail 2024-03-26T04:16:59.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:58 smithi183 ceph-mon[33745]: osdmap e54: 8 total, 8 up, 8 in 2024-03-26T04:16:59.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:58 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:16:59.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3074035808' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-26T04:16:59.992 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:59 smithi016 ceph-mon[29101]: osdmap e55: 8 total, 8 up, 8 in 2024-03-26T04:16:59.992 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:16:59 smithi016 ceph-mon[29101]: osdmap e56: 8 total, 8 up, 8 in 2024-03-26T04:16:59.993 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:59 smithi016 ceph-mon[37215]: osdmap e55: 8 total, 8 up, 8 in 2024-03-26T04:16:59.993 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:16:59 smithi016 ceph-mon[37215]: osdmap e56: 8 total, 8 up, 8 in 2024-03-26T04:17:00.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:59 smithi183 ceph-mon[33745]: osdmap e55: 8 total, 8 up, 8 in 2024-03-26T04:17:00.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:16:59 smithi183 ceph-mon[33745]: osdmap e56: 8 total, 8 up, 8 in 2024-03-26T04:17:00.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.521+0000 7ff0971f8700 1 -- 172.21.15.16:0/3669244533 >> v1:172.21.15.16:6789/0 conn(0x7ff09010acc0 legacy=0x7ff09010d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:00.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.521+0000 7ff0971f8700 1 -- 172.21.15.16:0/3669244533 shutdown_connections 2024-03-26T04:17:00.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.521+0000 7ff0971f8700 1 -- 172.21.15.16:0/3669244533 >> 172.21.15.16:0/3669244533 conn(0x7ff090077160 msgr2=0x7ff0900fe530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:17:00.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.521+0000 7ff0971f8700 1 -- 172.21.15.16:0/3669244533 shutdown_connections 2024-03-26T04:17:00.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.521+0000 7ff0971f8700 1 -- 172.21.15.16:0/3669244533 wait complete. 2024-03-26T04:17:00.523 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.522+0000 7ff0971f8700 1 Processor -- start 2024-03-26T04:17:00.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.522+0000 7ff0971f8700 1 -- start start 2024-03-26T04:17:00.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.522+0000 7ff0971f8700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff0901a8f80 con 0x7ff09010acc0 2024-03-26T04:17:00.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.522+0000 7ff0971f8700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff0901aa120 con 0x7ff09010e990 2024-03-26T04:17:00.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.522+0000 7ff0971f8700 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff0901ab2c0 con 0x7ff090106e40 2024-03-26T04:17:00.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.522+0000 7ff08ffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff09010acc0 0x7ff0901a4130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49584/0 (socket says 172.21.15.16:49584) 2024-03-26T04:17:00.524 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.522+0000 7ff08ffff700 1 -- 172.21.15.16:0/2672771940 learned_addr learned my addr 172.21.15.16:0/2672771940 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:17:00.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.522+0000 7ff08dffb700 1 -- 172.21.15.16:0/2672771940 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2902887506 0 0) 0x7ff0901ab2c0 con 0x7ff090106e40 2024-03-26T04:17:00.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.522+0000 7ff08dffb700 1 -- 172.21.15.16:0/2672771940 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0780049e0 con 0x7ff090106e40 2024-03-26T04:17:00.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.522+0000 7ff08dffb700 1 -- 172.21.15.16:0/2672771940 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2603766356 0 0) 0x7ff0901aa120 con 0x7ff09010e990 2024-03-26T04:17:00.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.522+0000 7ff08dffb700 1 -- 172.21.15.16:0/2672771940 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0901ab2c0 con 0x7ff09010e990 2024-03-26T04:17:00.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.522+0000 7ff08dffb700 1 -- 172.21.15.16:0/2672771940 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1059807318 0 0) 0x7ff0901a8f80 con 0x7ff09010acc0 2024-03-26T04:17:00.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.522+0000 7ff08dffb700 1 -- 172.21.15.16:0/2672771940 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0901aa120 con 0x7ff09010acc0 2024-03-26T04:17:00.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.522+0000 7ff08dffb700 1 -- 172.21.15.16:0/2672771940 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1161161154 0 0) 0x7ff0901ab2c0 con 0x7ff09010e990 2024-03-26T04:17:00.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.523+0000 7ff08dffb700 1 -- 172.21.15.16:0/2672771940 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0901a8f80 con 0x7ff09010e990 2024-03-26T04:17:00.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.523+0000 7ff08dffb700 1 -- 172.21.15.16:0/2672771940 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7ff088001590 con 0x7ff09010e990 2024-03-26T04:17:00.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.523+0000 7ff08dffb700 1 -- 172.21.15.16:0/2672771940 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3643406950 0 0) 0x7ff0780049e0 con 0x7ff090106e40 2024-03-26T04:17:00.526 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.523+0000 7ff08dffb700 1 -- 172.21.15.16:0/2672771940 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0901ab2c0 con 0x7ff090106e40 2024-03-26T04:17:00.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.523+0000 7ff08dffb700 1 -- 172.21.15.16:0/2672771940 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7ff0800017b0 con 0x7ff090106e40 2024-03-26T04:17:00.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.523+0000 7ff08dffb700 1 -- 172.21.15.16:0/2672771940 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3460292441 0 0) 0x7ff0901aa120 con 0x7ff09010acc0 2024-03-26T04:17:00.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.523+0000 7ff08dffb700 1 -- 172.21.15.16:0/2672771940 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff0780049e0 con 0x7ff09010acc0 2024-03-26T04:17:00.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.523+0000 7ff08dffb700 1 -- 172.21.15.16:0/2672771940 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7ff084004440 con 0x7ff09010acc0 2024-03-26T04:17:00.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.523+0000 7ff08dffb700 1 -- 172.21.15.16:0/2672771940 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1234025815 0 0) 0x7ff0901a8f80 con 0x7ff09010e990 2024-03-26T04:17:00.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.523+0000 7ff08dffb700 1 -- 172.21.15.16:0/2672771940 >> v1:172.21.15.16:6790/0 conn(0x7ff090106e40 legacy=0x7ff090102b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:00.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.523+0000 7ff08dffb700 1 -- 172.21.15.16:0/2672771940 >> v1:172.21.15.16:6789/0 conn(0x7ff09010acc0 legacy=0x7ff0901a4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:00.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.523+0000 7ff08dffb700 1 -- 172.21.15.16:0/2672771940 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0901ac4a0 con 0x7ff09010e990 2024-03-26T04:17:00.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.523+0000 7ff0971f8700 1 -- 172.21.15.16:0/2672771940 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0901aa310 con 0x7ff09010e990 2024-03-26T04:17:00.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.523+0000 7ff08dffb700 1 -- 172.21.15.16:0/2672771940 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7ff088003e90 con 0x7ff09010e990 2024-03-26T04:17:00.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.523+0000 7ff08dffb700 1 -- 172.21.15.16:0/2672771940 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7ff088004c40 con 0x7ff09010e990 2024-03-26T04:17:00.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.523+0000 7ff0971f8700 1 -- 172.21.15.16:0/2672771940 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0901aa790 con 0x7ff09010e990 2024-03-26T04:17:00.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.524+0000 7ff0971f8700 1 -- 172.21.15.16:0/2672771940 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff090103580 con 0x7ff09010e990 2024-03-26T04:17:00.527 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.524+0000 7ff08dffb700 1 -- 172.21.15.16:0/2672771940 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 15) v1 ==== 91363+0+0 (unknown 160314185 0 0) 0x7ff088003a00 con 0x7ff09010e990 2024-03-26T04:17:00.528 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.525+0000 7ff08dffb700 1 -- 172.21.15.16:0/2672771940 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(57..57 src has 1..57) v4 ==== 4081+0+0 (unknown 2092812373 0 0) 0x7ff088087140 con 0x7ff09010e990 2024-03-26T04:17:00.531 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.530+0000 7ff08dffb700 1 -- 172.21.15.16:0/2672771940 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7ff0880510a0 con 0x7ff09010e990 2024-03-26T04:17:00.739 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.738+0000 7ff0971f8700 1 -- 172.21.15.16:0/2672771940 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff090102620 con 0x7ff09010e990 2024-03-26T04:17:00.740 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.739+0000 7ff08dffb700 1 -- 172.21.15.16:0/2672771940 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+11478 (unknown 2588859899 0 101886682) 0x7ff088052fd0 con 0x7ff09010e990 2024-03-26T04:17:00.741 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:17:00.741 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":57,"fsid":"d7d59826-eb26-11ee-95d0-87774f69a715","created":"2024-03-26T04:11:49.624863+0000","modified":"2024-03-26T04:17:00.272918+0000","last_up_change":"2024-03-26T04:16:57.936663+0000","last_in_change":"2024-03-26T04:16:40.057008+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T04:14:58.099362+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e4e9d68-710d-4365-8810-e71718f37dee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":56,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6801","nonce":1132171550}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6802","nonce":1132171550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6804","nonce":1132171550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6803","nonce":1132171550}]},"public_addr":"172.21.15.16:6801/1132171550","cluster_addr":"172.21.15.16:6802/1132171550","heartbeat_back_addr":"172.21.15.16:6804/1132171550","heartbeat_front_addr":"172.21.15.16:6803/1132171550","state":["exists","up"]},{"osd":1,"uuid":"332e927f-79f4-4c33-b7f9-6d353d1721b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6805","nonce":317987982}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6806","nonce":317987982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6808","nonce":317987982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6807","nonce":317987982}]},"public_addr":"172.21.15.16:6805/317987982","cluster_addr":"172.21.15.16:6806/317987982","heartbeat_back_addr":"172.21.15.16:6808/317987982","heartbeat_front_addr":"172.21.15.16:6807/317987982","state":["exists","up"]},{"osd":2,"uuid":"b935affe-6e20-4226-87d9-12d4afe35679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6809","nonce":1107259766}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6810","nonce":1107259766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6812","nonce":1107259766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6811","nonce":1107259766}]},"public_addr":"172.21.15.16:6809/1107259766","cluster_addr":"172.21.15.16:6810/1107259766","heartbeat_back_addr":"172.21.15.16:6812/1107259766","heartbeat_front_addr":"172.21.15.16:6811/1107259766","state":["exists","up"]},{"osd":3,"uuid":"6c84a303-c00a-4a27-82c1-5e9ec1fb8d6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6813","nonce":1463789444}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6814","nonce":1463789444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6816","nonce":1463789444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6815","nonce":1463789444}]},"public_addr":"172.21.15.16:6813/1463789444","cluster_addr":"172.21.15.16:6814/1463789444","heartbeat_back_addr":"172.21.15.16:6816/1463789444","heartbeat_front_addr":"172.21.15.16:6815/1463789444","state":["exists","up"]},{"osd":4,"uuid":"750f2953-ae54-49a2-a428-9848615e9ddc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6800","nonce":879252532}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6801","nonce":879252532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6803","nonce":879252532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6802","nonce":879252532}]},"public_addr":"172.21.15.183:6800/879252532","cluster_addr":"172.21.15.183:6801/879252532","heartbeat_back_addr":"172.21.15.183:6803/879252532","heartbeat_front_addr":"172.21.15.183:6802/879252532","state":["exists","up"]},{"osd":5,"uuid":"0bbc822d-07fc-45c7-809d-a77e2cba6428","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6804","nonce":3356204259}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6805","nonce":3356204259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6807","nonce":3356204259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6806","nonce":3356204259}]},"public_addr":"172.21.15.183:6804/3356204259","cluster_addr":"172.21.15.183:6805/3356204259","heartbeat_back_addr":"172.21.15.183:6807/3356204259","heartbeat_front_addr":"172.21.15.183:6806/3356204259","state":["exists","up"]},{"osd":6,"uuid":"0de5f3f1-7a6e-4d11-9f99-db09de758531","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6808","nonce":777758685}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6809","nonce":777758685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6811","nonce":777758685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6810","nonce":777758685}]},"public_addr":"172.21.15.183:6808/777758685","cluster_addr":"172.21.15.183:6809/777758685","heartbeat_back_addr":"172.21.15.183:6811/777758685","heartbeat_front_addr":"172.21.15.183:6810/777758685","state":["exists","up"]},{"osd":7,"uuid":"6091e9e5-f241-4eed-81dd-970138c422c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6812","nonce":804876304}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6813","nonce":804876304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6815","nonce":804876304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6814","nonce":804876304}]},"public_addr":"172.21.15.183:6812/804876304","cluster_addr":"172.21.15.183:6813/804876304","heartbeat_back_addr":"172.21.15.183:6815/804876304","heartbeat_front_addr":"172.21.15.183:6814/804876304","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:14:05.613355+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:14:30.042974+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:14:55.072347+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:15:19.489278+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:15:42.363457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:16:07.018333+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:16:31.061483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:16:56.065175+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/285950115":"2024-03-27T04:12:43.759732+0000","172.21.15.16:0/292479134":"2024-03-27T04:12:43.759732+0000","172.21.15.16:6800/1828954920":"2024-03-27T04:12:43.759732+0000","172.21.15.16:0/1846690915":"2024-03-27T04:12:17.215822+0000","172.21.15.16:0/3092402688":"2024-03-27T04:12:17.215822+0000","172.21.15.16:0/1998037661":"2024-03-27T04:12:17.215822+0000","172.21.15.16:6800/3967690707":"2024-03-27T04:12:17.215822+0000","172.21.15.16:0/981571879":"2024-03-27T04:12:43.759732+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T04:17:00.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.741+0000 7ff0971f8700 1 -- 172.21.15.16:0/2672771940 >> v1:172.21.15.16:6800/4042489921 conn(0x7ff07806eae0 legacy=0x7ff078070fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:00.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.741+0000 7ff0971f8700 1 -- 172.21.15.16:0/2672771940 >> v1:172.21.15.183:6789/0 conn(0x7ff09010e990 legacy=0x7ff0901a7860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:00.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.742+0000 7ff0971f8700 1 -- 172.21.15.16:0/2672771940 shutdown_connections 2024-03-26T04:17:00.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.742+0000 7ff0971f8700 1 -- 172.21.15.16:0/2672771940 >> 172.21.15.16:0/2672771940 conn(0x7ff090077160 msgr2=0x7ff090111620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:17:00.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.742+0000 7ff0971f8700 1 -- 172.21.15.16:0/2672771940 shutdown_connections 2024-03-26T04:17:00.743 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:00.742+0000 7ff0971f8700 1 -- 172.21.15.16:0/2672771940 wait complete. 2024-03-26T04:17:01.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:01 smithi016 ceph-mon[37215]: pgmap v173: 1 pgs: 1 remapped+peering; 577 KiB data, 460 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:17:01.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:01 smithi016 ceph-mon[37215]: Detected new or changed devices on smithi183 2024-03-26T04:17:01.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:01 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:01.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:01 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:01.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:01 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:17:01.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:01 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:17:01.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:01 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:17:01.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:01 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:17:01.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:01 smithi016 ceph-mon[37215]: Adjusting osd_memory_target on smithi183 to 4272M 2024-03-26T04:17:01.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:01 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:01.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:01 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:17:01.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:01 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:17:01.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:01 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:01.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:01 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:17:01.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:01 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:17:01.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:01 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:17:01.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:01 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:01.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:01 smithi016 ceph-mon[37215]: osdmap e57: 8 total, 8 up, 8 in 2024-03-26T04:17:01.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:01 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2672771940' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:17:01.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:01 smithi016 ceph-mon[29101]: pgmap v173: 1 pgs: 1 remapped+peering; 577 KiB data, 460 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:17:01.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:01 smithi016 ceph-mon[29101]: Detected new or changed devices on smithi183 2024-03-26T04:17:01.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:01 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:01.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:01 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:01.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:01 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:17:01.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:01 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:17:01.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:01 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:17:01.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:01 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:17:01.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:01 smithi016 ceph-mon[29101]: Adjusting osd_memory_target on smithi183 to 4272M 2024-03-26T04:17:01.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:01 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:01.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:01 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:17:01.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:01 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:17:01.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:01 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:01.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:01 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:17:01.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:01 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:17:01.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:01 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:17:01.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:01 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:01.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:01 smithi016 ceph-mon[29101]: osdmap e57: 8 total, 8 up, 8 in 2024-03-26T04:17:01.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:01 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2672771940' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:17:01.469 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-26T04:14:58.099362+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-26T04:17:01.469 DEBUG:teuthology.orchestra.run.smithi016:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-26T04:17:01.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:01 smithi183 ceph-mon[33745]: pgmap v173: 1 pgs: 1 remapped+peering; 577 KiB data, 460 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:17:01.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:01 smithi183 ceph-mon[33745]: Detected new or changed devices on smithi183 2024-03-26T04:17:01.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:01 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:01.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:01 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:01.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:01 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:17:01.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:01 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:17:01.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:01 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:17:01.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:01 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:17:01.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:01 smithi183 ceph-mon[33745]: Adjusting osd_memory_target on smithi183 to 4272M 2024-03-26T04:17:01.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:01 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:01.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:01 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:17:01.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:01 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:17:01.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:01 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:01.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:01 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:17:01.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:01 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:17:01.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:01 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:17:01.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:01 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:01.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:01 smithi183 ceph-mon[33745]: osdmap e57: 8 total, 8 up, 8 in 2024-03-26T04:17:01.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:01 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2672771940' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:17:01.714 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:17:02.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:02 smithi016 ceph-mon[29101]: osdmap e58: 8 total, 8 up, 8 in 2024-03-26T04:17:02.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:02 smithi016 ceph-mon[37215]: osdmap e58: 8 total, 8 up, 8 in 2024-03-26T04:17:02.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:02 smithi183 ceph-mon[33745]: osdmap e58: 8 total, 8 up, 8 in 2024-03-26T04:17:03.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.162+0000 7f3ab73ba700 1 -- 172.21.15.16:0/1266230285 >> v1:172.21.15.16:6790/0 conn(0x7f3ab0103030 legacy=0x7f3ab0103410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:03.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.162+0000 7f3ab73ba700 1 -- 172.21.15.16:0/1266230285 shutdown_connections 2024-03-26T04:17:03.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.162+0000 7f3ab73ba700 1 -- 172.21.15.16:0/1266230285 >> 172.21.15.16:0/1266230285 conn(0x7f3ab00fc820 msgr2=0x7f3ab00fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:17:03.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.162+0000 7f3ab73ba700 1 -- 172.21.15.16:0/1266230285 shutdown_connections 2024-03-26T04:17:03.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.162+0000 7f3ab73ba700 1 -- 172.21.15.16:0/1266230285 wait complete. 2024-03-26T04:17:03.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.163+0000 7f3ab73ba700 1 Processor -- start 2024-03-26T04:17:03.163 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.163+0000 7f3ab73ba700 1 -- start start 2024-03-26T04:17:03.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.163+0000 7f3ab73ba700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3ab01a6c20 con 0x7f3ab0103030 2024-03-26T04:17:03.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.163+0000 7f3ab73ba700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3ab01a7e00 con 0x7f3ab010ae40 2024-03-26T04:17:03.165 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.163+0000 7f3ab73ba700 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3ab01a9000 con 0x7f3ab010e950 2024-03-26T04:17:03.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.163+0000 7f3ab5156700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3ab0103030 0x7f3ab010a470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49608/0 (socket says 172.21.15.16:49608) 2024-03-26T04:17:03.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.163+0000 7f3ab5156700 1 -- 172.21.15.16:0/4111315628 learned_addr learned my addr 172.21.15.16:0/4111315628 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:17:03.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.163+0000 7f3aa67fc700 1 -- 172.21.15.16:0/4111315628 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3623709570 0 0) 0x7f3ab01a9000 con 0x7f3ab010e950 2024-03-26T04:17:03.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.163+0000 7f3aa67fc700 1 -- 172.21.15.16:0/4111315628 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a9c0031f0 con 0x7f3ab010e950 2024-03-26T04:17:03.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.163+0000 7f3aa67fc700 1 -- 172.21.15.16:0/4111315628 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1953730180 0 0) 0x7f3ab01a6c20 con 0x7f3ab0103030 2024-03-26T04:17:03.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.163+0000 7f3aa67fc700 1 -- 172.21.15.16:0/4111315628 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ab01a9000 con 0x7f3ab0103030 2024-03-26T04:17:03.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.163+0000 7f3aa67fc700 1 -- 172.21.15.16:0/4111315628 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1536118323 0 0) 0x7f3ab01a7e00 con 0x7f3ab010ae40 2024-03-26T04:17:03.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.163+0000 7f3aa67fc700 1 -- 172.21.15.16:0/4111315628 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ab01a6c20 con 0x7f3ab010ae40 2024-03-26T04:17:03.166 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.164+0000 7f3aa67fc700 1 -- 172.21.15.16:0/4111315628 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3433419975 0 0) 0x7f3a9c0031f0 con 0x7f3ab010e950 2024-03-26T04:17:03.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.164+0000 7f3aa67fc700 1 -- 172.21.15.16:0/4111315628 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ab01a7e00 con 0x7f3ab010e950 2024-03-26T04:17:03.167 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.164+0000 7f3aa67fc700 1 -- 172.21.15.16:0/4111315628 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f3aa8001b70 con 0x7f3ab010e950 2024-03-26T04:17:03.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.164+0000 7f3aa67fc700 1 -- 172.21.15.16:0/4111315628 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 809758294 0 0) 0x7f3ab01a9000 con 0x7f3ab0103030 2024-03-26T04:17:03.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.164+0000 7f3aa67fc700 1 -- 172.21.15.16:0/4111315628 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3a9c0031f0 con 0x7f3ab0103030 2024-03-26T04:17:03.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.164+0000 7f3aa67fc700 1 -- 172.21.15.16:0/4111315628 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f3aac003860 con 0x7f3ab0103030 2024-03-26T04:17:03.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.164+0000 7f3aa67fc700 1 -- 172.21.15.16:0/4111315628 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1931013137 0 0) 0x7f3ab01a6c20 con 0x7f3ab010ae40 2024-03-26T04:17:03.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.164+0000 7f3aa67fc700 1 -- 172.21.15.16:0/4111315628 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ab01a9000 con 0x7f3ab010ae40 2024-03-26T04:17:03.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.164+0000 7f3aa67fc700 1 -- 172.21.15.16:0/4111315628 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f3aa00019f0 con 0x7f3ab010ae40 2024-03-26T04:17:03.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.164+0000 7f3aa67fc700 1 -- 172.21.15.16:0/4111315628 <== mon.2 v1:172.21.15.16:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3283801040 0 0) 0x7f3ab01a7e00 con 0x7f3ab010e950 2024-03-26T04:17:03.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.164+0000 7f3aa67fc700 1 -- 172.21.15.16:0/4111315628 >> v1:172.21.15.183:6789/0 conn(0x7f3ab010ae40 legacy=0x7f3ab01a1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:03.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.164+0000 7f3aa67fc700 1 -- 172.21.15.16:0/4111315628 >> v1:172.21.15.16:6789/0 conn(0x7f3ab0103030 legacy=0x7f3ab010a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:03.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.164+0000 7f3aa67fc700 1 -- 172.21.15.16:0/4111315628 --> v1:172.21.15.16:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ab01aa200 con 0x7f3ab010e950 2024-03-26T04:17:03.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.164+0000 7f3ab73ba700 1 -- 172.21.15.16:0/4111315628 --> v1:172.21.15.16:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ab01a6db0 con 0x7f3ab010e950 2024-03-26T04:17:03.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.164+0000 7f3ab73ba700 1 -- 172.21.15.16:0/4111315628 --> v1:172.21.15.16:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ab01a7230 con 0x7f3ab010e950 2024-03-26T04:17:03.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.164+0000 7f3aa67fc700 1 -- 172.21.15.16:0/4111315628 <== mon.2 v1:172.21.15.16:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f3aa8003e90 con 0x7f3ab010e950 2024-03-26T04:17:03.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.164+0000 7f3aa67fc700 1 -- 172.21.15.16:0/4111315628 <== mon.2 v1:172.21.15.16:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f3aa8004db0 con 0x7f3ab010e950 2024-03-26T04:17:03.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.165+0000 7f3ab73ba700 1 -- 172.21.15.16:0/4111315628 --> v1:172.21.15.16:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a94005120 con 0x7f3ab010e950 2024-03-26T04:17:03.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.165+0000 7f3aa67fc700 1 -- 172.21.15.16:0/4111315628 <== mon.2 v1:172.21.15.16:6790/0 7 ==== mgrmap(e 15) v1 ==== 91363+0+0 (unknown 160314185 0 0) 0x7f3aa8003a00 con 0x7f3ab010e950 2024-03-26T04:17:03.169 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.166+0000 7f3aa67fc700 1 -- 172.21.15.16:0/4111315628 <== mon.2 v1:172.21.15.16:6790/0 8 ==== osd_map(58..58 src has 1..58) v4 ==== 4081+0+0 (unknown 3634163543 0 0) 0x7f3aa80871e0 con 0x7f3ab010e950 2024-03-26T04:17:03.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.171+0000 7f3aa67fc700 1 -- 172.21.15.16:0/4111315628 <== mon.2 v1:172.21.15.16:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f3aa80511a0 con 0x7f3ab010e950 2024-03-26T04:17:03.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.376+0000 7f3ab73ba700 1 -- 172.21.15.16:0/4111315628 --> v1:172.21.15.16:6790/0 -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3a94005950 con 0x7f3ab010e950 2024-03-26T04:17:03.376 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.376+0000 7f3aa67fc700 1 -- 172.21.15.16:0/4111315628 <== mon.2 v1:172.21.15.16:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v58) v1 ==== 93+0+10 (unknown 3844627101 0 2170607528) 0x7f3aa80530d0 con 0x7f3ab010e950 2024-03-26T04:17:03.377 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 1 2024-03-26T04:17:03.378 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.378+0000 7f3ab73ba700 1 -- 172.21.15.16:0/4111315628 >> v1:172.21.15.16:6800/4042489921 conn(0x7f3a9c06e9d0 legacy=0x7f3a9c070e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:03.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.378+0000 7f3ab73ba700 1 -- 172.21.15.16:0/4111315628 >> v1:172.21.15.16:6790/0 conn(0x7f3ab010e950 legacy=0x7f3ab01a5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:03.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.378+0000 7f3ab73ba700 1 -- 172.21.15.16:0/4111315628 shutdown_connections 2024-03-26T04:17:03.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.378+0000 7f3ab73ba700 1 -- 172.21.15.16:0/4111315628 >> 172.21.15.16:0/4111315628 conn(0x7f3ab00fc820 msgr2=0x7f3ab00fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:17:03.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.378+0000 7f3ab73ba700 1 -- 172.21.15.16:0/4111315628 shutdown_connections 2024-03-26T04:17:03.379 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:03.378+0000 7f3ab73ba700 1 -- 172.21.15.16:0/4111315628 wait complete. 2024-03-26T04:17:03.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:03 smithi183 ceph-mon[33745]: pgmap v177: 1 pgs: 1 remapped+peering; 577 KiB data, 460 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:17:03.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:03 smithi016 ceph-mon[29101]: pgmap v177: 1 pgs: 1 remapped+peering; 577 KiB data, 460 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:17:03.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:03 smithi016 ceph-mon[37215]: pgmap v177: 1 pgs: 1 remapped+peering; 577 KiB data, 460 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:17:03.985 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi016 2024-03-26T04:17:03.985 DEBUG:teuthology.orchestra.run.smithi183:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi016=foo.a' 2024-03-26T04:17:04.234 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:17:04.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:04 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/4111315628' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T04:17:04.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:04 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/4111315628' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T04:17:04.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:04 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/4111315628' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-26T04:17:05.371 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:05 smithi183 ceph-mon[33745]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-03-26T04:17:05.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:05 smithi016 ceph-mon[29101]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-03-26T04:17:05.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:05 smithi016 ceph-mon[37215]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-03-26T04:17:05.890 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:05.884+0000 7f23d6190700 1 -- 172.21.15.183:0/2138319373 >> v1:172.21.15.183:6789/0 conn(0x7f23d01051b0 legacy=0x7f23d0107600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:05.891 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:05.884+0000 7f23d6190700 1 -- 172.21.15.183:0/2138319373 shutdown_connections 2024-03-26T04:17:05.891 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:05.884+0000 7f23d6190700 1 -- 172.21.15.183:0/2138319373 >> 172.21.15.183:0/2138319373 conn(0x7f23d00fc360 msgr2=0x7f23d00fe780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:17:05.891 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:05.884+0000 7f23d6190700 1 -- 172.21.15.183:0/2138319373 shutdown_connections 2024-03-26T04:17:05.891 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:05.884+0000 7f23d6190700 1 -- 172.21.15.183:0/2138319373 wait complete. 2024-03-26T04:17:05.891 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:05.885+0000 7f23d6190700 1 Processor -- start 2024-03-26T04:17:05.891 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:05.885+0000 7f23d6190700 1 -- start start 2024-03-26T04:17:05.892 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:05.885+0000 7f23d6190700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f23d0110000 con 0x7f23d0108ef0 2024-03-26T04:17:05.892 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:05.885+0000 7f23d6190700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f23d0196cb0 con 0x7f23d01051b0 2024-03-26T04:17:05.892 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:05.885+0000 7f23d6190700 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f23d01a3450 con 0x7f23d0100d80 2024-03-26T04:17:05.892 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:05.885+0000 7f23ceffd700 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f23d01051b0 0x7f23d019c5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:40486/0 (socket says 172.21.15.183:40486) 2024-03-26T04:17:05.892 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:05.885+0000 7f23ceffd700 1 -- 172.21.15.183:0/3129885703 learned_addr learned my addr 172.21.15.183:0/3129885703 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-26T04:17:05.892 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:05.886+0000 7f23ccff9700 1 -- 172.21.15.183:0/3129885703 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 408155588 0 0) 0x7f23d0196cb0 con 0x7f23d01051b0 2024-03-26T04:17:05.893 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:05.886+0000 7f23ccff9700 1 -- 172.21.15.183:0/3129885703 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23b80037f0 con 0x7f23d01051b0 2024-03-26T04:17:05.893 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:05.886+0000 7f23ccff9700 1 -- 172.21.15.183:0/3129885703 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3329064609 0 0) 0x7f23d01a3450 con 0x7f23d0100d80 2024-03-26T04:17:05.893 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:05.886+0000 7f23ccff9700 1 -- 172.21.15.183:0/3129885703 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23d0196cb0 con 0x7f23d0100d80 2024-03-26T04:17:05.893 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:05.886+0000 7f23ccff9700 1 -- 172.21.15.183:0/3129885703 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 212310451 0 0) 0x7f23d0110000 con 0x7f23d0108ef0 2024-03-26T04:17:05.893 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:05.886+0000 7f23ccff9700 1 -- 172.21.15.183:0/3129885703 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23d01a3450 con 0x7f23d0108ef0 2024-03-26T04:17:05.893 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:05.886+0000 7f23ccff9700 1 -- 172.21.15.183:0/3129885703 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3781255799 0 0) 0x7f23b80037f0 con 0x7f23d01051b0 2024-03-26T04:17:05.894 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:05.886+0000 7f23ccff9700 1 -- 172.21.15.183:0/3129885703 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23d0110000 con 0x7f23d01051b0 2024-03-26T04:17:05.894 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:05.886+0000 7f23ccff9700 1 -- 172.21.15.183:0/3129885703 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f23c4004460 con 0x7f23d01051b0 2024-03-26T04:17:05.894 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:05.886+0000 7f23ccff9700 1 -- 172.21.15.183:0/3129885703 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1371755052 0 0) 0x7f23d01a3450 con 0x7f23d0108ef0 2024-03-26T04:17:05.894 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:05.886+0000 7f23ccff9700 1 -- 172.21.15.183:0/3129885703 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23b80037f0 con 0x7f23d0108ef0 2024-03-26T04:17:05.894 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:05.886+0000 7f23ccff9700 1 -- 172.21.15.183:0/3129885703 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2268201243 0 0) 0x7f23d0196cb0 con 0x7f23d0100d80 2024-03-26T04:17:05.894 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:05.886+0000 7f23ccff9700 1 -- 172.21.15.183:0/3129885703 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23d01a3450 con 0x7f23d0100d80 2024-03-26T04:17:05.895 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:05.886+0000 7f23ccff9700 1 -- 172.21.15.183:0/3129885703 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f23c80018b0 con 0x7f23d0108ef0 2024-03-26T04:17:05.895 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:05.886+0000 7f23ccff9700 1 -- 172.21.15.183:0/3129885703 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f23c0001b50 con 0x7f23d0100d80 2024-03-26T04:17:05.895 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:05.886+0000 7f23ccff9700 1 -- 172.21.15.183:0/3129885703 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1127875353 0 0) 0x7f23d0110000 con 0x7f23d01051b0 2024-03-26T04:17:05.895 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:05.886+0000 7f23ccff9700 1 -- 172.21.15.183:0/3129885703 >> v1:172.21.15.16:6790/0 conn(0x7f23d0100d80 legacy=0x7f23d0196100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:05.895 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:05.886+0000 7f23ccff9700 1 -- 172.21.15.183:0/3129885703 >> v1:172.21.15.16:6789/0 conn(0x7f23d0108ef0 legacy=0x7f23d019fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:05.896 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:05.886+0000 7f23ccff9700 1 -- 172.21.15.183:0/3129885703 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d01a45f0 con 0x7f23d01051b0 2024-03-26T04:17:05.896 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:05.886+0000 7f23d6190700 1 -- 172.21.15.183:0/3129885703 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23d01a3640 con 0x7f23d01051b0 2024-03-26T04:17:05.896 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:05.886+0000 7f23d6190700 1 -- 172.21.15.183:0/3129885703 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f23d01a3b40 con 0x7f23d01051b0 2024-03-26T04:17:05.896 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:05.887+0000 7f23ccff9700 1 -- 172.21.15.183:0/3129885703 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f23c40046c0 con 0x7f23d01051b0 2024-03-26T04:17:05.896 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:05.887+0000 7f23ccff9700 1 -- 172.21.15.183:0/3129885703 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f23c4004eb0 con 0x7f23d01051b0 2024-03-26T04:17:05.897 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:05.888+0000 7f23d6190700 1 -- 172.21.15.183:0/3129885703 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23bc005150 con 0x7f23d01051b0 2024-03-26T04:17:05.897 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:05.888+0000 7f23ccff9700 1 -- 172.21.15.183:0/3129885703 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 15) v1 ==== 91363+0+0 (unknown 160314185 0 0) 0x7f23c4005010 con 0x7f23d01051b0 2024-03-26T04:17:05.897 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:05.888+0000 7f23ccff9700 1 -- 172.21.15.183:0/3129885703 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(58..58 src has 1..58) v4 ==== 4081+0+0 (unknown 3634163543 0 0) 0x7f23c4088240 con 0x7f23d01051b0 2024-03-26T04:17:05.898 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:05.893+0000 7f23ccff9700 1 -- 172.21.15.183:0/3129885703 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f23c40521a0 con 0x7f23d01051b0 2024-03-26T04:17:06.101 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:06.101+0000 7f23d6190700 1 -- 172.21.15.183:0/3129885703 --> v1:172.21.15.16:6800/4042489921 -- mgr_command(tid 0: {"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi016=foo.a", "target": ["mon-mgr", ""]}) v1 -- 0x7f23bc000bc0 con 0x7f23b806ed20 2024-03-26T04:17:06.108 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:06.108+0000 7f23ccff9700 1 -- 172.21.15.183:0/3129885703 <== mgr.14152 v1:172.21.15.16:6800/4042489921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+30 (unknown 0 0 1123153589) 0x7f23bc000bc0 con 0x7f23b806ed20 2024-03-26T04:17:06.109 INFO:teuthology.orchestra.run.smithi183.stdout:Scheduled rgw.foo.a update... 2024-03-26T04:17:06.111 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:06.110+0000 7f23d6190700 1 -- 172.21.15.183:0/3129885703 >> v1:172.21.15.16:6800/4042489921 conn(0x7f23b806ed20 legacy=0x7f23b80711e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:06.111 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:06.110+0000 7f23d6190700 1 -- 172.21.15.183:0/3129885703 >> v1:172.21.15.183:6789/0 conn(0x7f23d01051b0 legacy=0x7f23d019c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:06.111 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:06.110+0000 7f23d6190700 1 -- 172.21.15.183:0/3129885703 shutdown_connections 2024-03-26T04:17:06.111 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:06.110+0000 7f23d6190700 1 -- 172.21.15.183:0/3129885703 >> 172.21.15.183:0/3129885703 conn(0x7f23d00fc360 msgr2=0x7f23d0107600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:17:06.111 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:06.110+0000 7f23d6190700 1 -- 172.21.15.183:0/3129885703 shutdown_connections 2024-03-26T04:17:06.111 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:06.110+0000 7f23d6190700 1 -- 172.21.15.183:0/3129885703 wait complete. 2024-03-26T04:17:06.757 DEBUG:teuthology.orchestra.run.smithi016:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-d7d59826-eb26-11ee-95d0-87774f69a715@rgw.foo.a.service 2024-03-26T04:17:06.759 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi183 2024-03-26T04:17:06.760 DEBUG:teuthology.orchestra.run.smithi183:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph osd pool create datapool 3 3 replicated 2024-03-26T04:17:06.995 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:17:07.371 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:07 smithi016 ceph-mon[37215]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T04:17:07.372 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:07 smithi016 ceph-mon[37215]: from='client.24352 v1:172.21.15.183:0/3129885703' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi016=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:17:07.372 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:07 smithi016 ceph-mon[37215]: Saving service rgw.foo.a spec with placement smithi016=foo.a;count:1 2024-03-26T04:17:07.372 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:07 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:07.372 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:07 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:17:07.372 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:07 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:17:07.373 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:07 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:17:07.373 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:07 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:07.373 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:07 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:17:07.373 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:07 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T04:17:07.373 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:07 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:07.373 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:07 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:17:07.373 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:07 smithi016 ceph-mon[37215]: Deploying daemon rgw.foo.a on smithi016 2024-03-26T04:17:07.374 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:07 smithi016 ceph-mon[29101]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T04:17:07.374 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:07 smithi016 ceph-mon[29101]: from='client.24352 v1:172.21.15.183:0/3129885703' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi016=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:17:07.374 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:07 smithi016 ceph-mon[29101]: Saving service rgw.foo.a spec with placement smithi016=foo.a;count:1 2024-03-26T04:17:07.374 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:07 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:07.374 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:07 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:17:07.375 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:07 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:17:07.375 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:07 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:17:07.375 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:07 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:07.375 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:07 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:17:07.375 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:07 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T04:17:07.375 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:07 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:07.375 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:07 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:17:07.376 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:07 smithi016 ceph-mon[29101]: Deploying daemon rgw.foo.a on smithi016 2024-03-26T04:17:07.431 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:07 smithi183 ceph-mon[33745]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-26T04:17:07.431 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:07 smithi183 ceph-mon[33745]: from='client.24352 v1:172.21.15.183:0/3129885703' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi016=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:17:07.431 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:07 smithi183 ceph-mon[33745]: Saving service rgw.foo.a spec with placement smithi016=foo.a;count:1 2024-03-26T04:17:07.431 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:07 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:07.431 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:07 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:17:07.431 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:07 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:17:07.431 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:07 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:17:07.431 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:07 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:07.431 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:07 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-26T04:17:07.431 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:07 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-26T04:17:07.431 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:07 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:07.432 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:07 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:17:07.432 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:07 smithi183 ceph-mon[33745]: Deploying daemon rgw.foo.a on smithi016 2024-03-26T04:17:08.553 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:08.550+0000 7f4024fe4700 1 -- 172.21.15.183:0/3881341434 >> v1:172.21.15.183:6789/0 conn(0x7f4020101e90 legacy=0x7f40201022b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:08.553 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:08.550+0000 7f4024fe4700 1 -- 172.21.15.183:0/3881341434 shutdown_connections 2024-03-26T04:17:08.553 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:08.550+0000 7f4024fe4700 1 -- 172.21.15.183:0/3881341434 >> 172.21.15.183:0/3881341434 conn(0x7f40200fd430 msgr2=0x7f40200ff870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:17:08.553 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:08.550+0000 7f4024fe4700 1 -- 172.21.15.183:0/3881341434 shutdown_connections 2024-03-26T04:17:08.553 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:08.550+0000 7f4024fe4700 1 -- 172.21.15.183:0/3881341434 wait complete. 2024-03-26T04:17:08.553 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:08.550+0000 7f4024fe4700 1 Processor -- start 2024-03-26T04:17:08.553 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:08.551+0000 7f4024fe4700 1 -- start start 2024-03-26T04:17:08.553 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:08.551+0000 7f4024fe4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f4020197d60 con 0x7f4020109f40 2024-03-26T04:17:08.554 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:08.551+0000 7f4024fe4700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f40201a3550 con 0x7f40201062b0 2024-03-26T04:17:08.554 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:08.551+0000 7f4024fe4700 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f40201a46f0 con 0x7f4020101e90 2024-03-26T04:17:08.554 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:08.551+0000 7f401dd9b700 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f40201062b0 0x7f402019d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:35238/0 (socket says 172.21.15.183:35238) 2024-03-26T04:17:08.554 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:08.551+0000 7f401dd9b700 1 -- 172.21.15.183:0/827055484 learned_addr learned my addr 172.21.15.183:0/827055484 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-26T04:17:08.554 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:08.551+0000 7f400f7fe700 1 -- 172.21.15.183:0/827055484 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1392177984 0 0) 0x7f40201a3550 con 0x7f40201062b0 2024-03-26T04:17:08.554 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:08.551+0000 7f400f7fe700 1 -- 172.21.15.183:0/827055484 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40080035b0 con 0x7f40201062b0 2024-03-26T04:17:08.554 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:08.551+0000 7f400f7fe700 1 -- 172.21.15.183:0/827055484 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 887858767 0 0) 0x7f40201a46f0 con 0x7f4020101e90 2024-03-26T04:17:08.554 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:08.551+0000 7f400f7fe700 1 -- 172.21.15.183:0/827055484 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40201a3550 con 0x7f4020101e90 2024-03-26T04:17:08.554 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:08.551+0000 7f400f7fe700 1 -- 172.21.15.183:0/827055484 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4206485559 0 0) 0x7f4020197d60 con 0x7f4020109f40 2024-03-26T04:17:08.554 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:08.551+0000 7f400f7fe700 1 -- 172.21.15.183:0/827055484 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40201a46f0 con 0x7f4020109f40 2024-03-26T04:17:08.554 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:08.551+0000 7f400f7fe700 1 -- 172.21.15.183:0/827055484 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 351383395 0 0) 0x7f40080035b0 con 0x7f40201062b0 2024-03-26T04:17:08.554 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:08.551+0000 7f400f7fe700 1 -- 172.21.15.183:0/827055484 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4020197d60 con 0x7f40201062b0 2024-03-26T04:17:08.554 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:08.551+0000 7f400f7fe700 1 -- 172.21.15.183:0/827055484 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f40100018b0 con 0x7f40201062b0 2024-03-26T04:17:08.555 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:08.551+0000 7f400f7fe700 1 -- 172.21.15.183:0/827055484 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4230134517 0 0) 0x7f40201a3550 con 0x7f4020101e90 2024-03-26T04:17:08.555 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:08.551+0000 7f400f7fe700 1 -- 172.21.15.183:0/827055484 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40080035b0 con 0x7f4020101e90 2024-03-26T04:17:08.555 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:08.551+0000 7f400f7fe700 1 -- 172.21.15.183:0/827055484 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f4014001880 con 0x7f4020101e90 2024-03-26T04:17:08.555 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:08.551+0000 7f400f7fe700 1 -- 172.21.15.183:0/827055484 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 680216677 0 0) 0x7f40201a46f0 con 0x7f4020109f40 2024-03-26T04:17:08.555 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:08.551+0000 7f400f7fe700 1 -- 172.21.15.183:0/827055484 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40201a3550 con 0x7f4020109f40 2024-03-26T04:17:08.555 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:08.551+0000 7f400f7fe700 1 -- 172.21.15.183:0/827055484 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f4018001c30 con 0x7f4020109f40 2024-03-26T04:17:08.555 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:08.552+0000 7f400f7fe700 1 -- 172.21.15.183:0/827055484 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2791940899 0 0) 0x7f4020197d60 con 0x7f40201062b0 2024-03-26T04:17:08.555 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:08.552+0000 7f400f7fe700 1 -- 172.21.15.183:0/827055484 >> v1:172.21.15.16:6790/0 conn(0x7f4020101e90 legacy=0x7f4020197210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:08.555 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:08.552+0000 7f400f7fe700 1 -- 172.21.15.183:0/827055484 >> v1:172.21.15.16:6789/0 conn(0x7f4020109f40 legacy=0x7f40201a0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:08.555 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:08.552+0000 7f400f7fe700 1 -- 172.21.15.183:0/827055484 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40201a5890 con 0x7f40201062b0 2024-03-26T04:17:08.555 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:08.552+0000 7f4024fe4700 1 -- 172.21.15.183:0/827055484 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40201a48e0 con 0x7f40201062b0 2024-03-26T04:17:08.555 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:08.552+0000 7f4024fe4700 1 -- 172.21.15.183:0/827055484 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f40201a4de0 con 0x7f40201062b0 2024-03-26T04:17:08.555 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:08.552+0000 7f400f7fe700 1 -- 172.21.15.183:0/827055484 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f4010003b30 con 0x7f40201062b0 2024-03-26T04:17:08.555 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:08.552+0000 7f400f7fe700 1 -- 172.21.15.183:0/827055484 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f4010005b70 con 0x7f40201062b0 2024-03-26T04:17:08.555 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:08.553+0000 7f4024fe4700 1 -- 172.21.15.183:0/827055484 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4000005150 con 0x7f40201062b0 2024-03-26T04:17:08.556 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:08.553+0000 7f400f7fe700 1 -- 172.21.15.183:0/827055484 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 15) v1 ==== 91363+0+0 (unknown 160314185 0 0) 0x7f40100036a0 con 0x7f40201062b0 2024-03-26T04:17:08.556 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:08.553+0000 7f400f7fe700 1 -- 172.21.15.183:0/827055484 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(58..58 src has 1..58) v4 ==== 4081+0+0 (unknown 3634163543 0 0) 0x7f4010087d40 con 0x7f40201062b0 2024-03-26T04:17:08.559 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:08.558+0000 7f400f7fe700 1 -- 172.21.15.183:0/827055484 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f4010051de0 con 0x7f40201062b0 2024-03-26T04:17:08.752 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:08.751+0000 7f4024fe4700 1 -- 172.21.15.183:0/827055484 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"} v 0) v1 -- 0x7f4000005980 con 0x7f40201062b0 2024-03-26T04:17:09.116 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:09.116+0000 7f400f7fe700 1 -- 172.21.15.183:0/827055484 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]=0 pool 'datapool' created v59) v1 ==== 160+0+0 (unknown 1751278495 0 0) 0x7f4010053d10 con 0x7f40201062b0 2024-03-26T04:17:09.118 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:09.118+0000 7f4024fe4700 1 -- 172.21.15.183:0/827055484 >> v1:172.21.15.16:6800/4042489921 conn(0x7f400806e940 legacy=0x7f4008070e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:09.118 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:09.118+0000 7f4024fe4700 1 -- 172.21.15.183:0/827055484 >> v1:172.21.15.183:6789/0 conn(0x7f40201062b0 legacy=0x7f402019d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:09.119 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:09.118+0000 7f4024fe4700 1 -- 172.21.15.183:0/827055484 shutdown_connections 2024-03-26T04:17:09.119 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:09.118+0000 7f4024fe4700 1 -- 172.21.15.183:0/827055484 >> 172.21.15.183:0/827055484 conn(0x7f40200fd430 msgr2=0x7f402010c400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:17:09.119 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:09.118+0000 7f4024fe4700 1 -- 172.21.15.183:0/827055484 shutdown_connections 2024-03-26T04:17:09.119 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:09.118+0000 7f4024fe4700 1 -- 172.21.15.183:0/827055484 wait complete. 2024-03-26T04:17:09.119 INFO:teuthology.orchestra.run.smithi183.stderr:pool 'datapool' created 2024-03-26T04:17:09.248 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:09 smithi016 ceph-mon[29101]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-26T04:17:09.248 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:09 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.183:0/827055484' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T04:17:09.248 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:09 smithi016 ceph-mon[29101]: from='client.24358 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T04:17:09.248 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:09 smithi016 ceph-mon[37215]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-26T04:17:09.249 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:09 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.183:0/827055484' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T04:17:09.249 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:09 smithi016 ceph-mon[37215]: from='client.24358 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T04:17:09.249 INFO:journalctl@ceph.rgw.foo.a.smithi016.stdout:Mar 26 04:17:09 smithi016 podman[63901]: 2024-03-26 04:17:09.245803754 +0000 UTC m=+0.367470589 container init df20b4bd81a97ca97f72f6895b3614c0e8d150782cebd68c166eb08cabb40ca8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-rgw-foo-a, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-26T04:17:09.249 INFO:journalctl@ceph.rgw.foo.a.smithi016.stdout:Mar 26 04:17:09 smithi016 podman[63901]: 2024-03-26 04:17:09.248412293 +0000 UTC m=+0.370079128 container start df20b4bd81a97ca97f72f6895b3614c0e8d150782cebd68c166eb08cabb40ca8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-rgw-foo-a, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T04:17:09.432 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:09 smithi183 ceph-mon[33745]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-03-26T04:17:09.432 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:09 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.183:0/827055484' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T04:17:09.432 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:09 smithi183 ceph-mon[33745]: from='client.24358 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-26T04:17:09.662 INFO:journalctl@ceph.rgw.foo.a.smithi016.stdout:Mar 26 04:17:09 smithi016 bash[63901]: df20b4bd81a97ca97f72f6895b3614c0e8d150782cebd68c166eb08cabb40ca8 2024-03-26T04:17:09.662 INFO:journalctl@ceph.rgw.foo.a.smithi016.stdout:Mar 26 04:17:09 smithi016 systemd[1]: Started Ceph rgw.foo.a for d7d59826-eb26-11ee-95d0-87774f69a715. 2024-03-26T04:17:09.768 DEBUG:teuthology.orchestra.run.smithi183:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- rbd pool init datapool 2024-03-26T04:17:10.003 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:17:10.366 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:10 smithi183 ceph-mon[33745]: from='client.24358 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-26T04:17:10.366 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:10 smithi183 ceph-mon[33745]: osdmap e59: 8 total, 8 up, 8 in 2024-03-26T04:17:10.366 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:10 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:10.367 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:10 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:10.367 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:10 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:10.367 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:10 smithi183 ceph-mon[33745]: Saving service rgw.foo.a spec with placement smithi016=foo.a;count:1 2024-03-26T04:17:10.367 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:10 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:10.367 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:10 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:10.367 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:10 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:17:10.400 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:10 smithi016 ceph-mon[29101]: from='client.24358 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-26T04:17:10.400 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:10 smithi016 ceph-mon[29101]: osdmap e59: 8 total, 8 up, 8 in 2024-03-26T04:17:10.401 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:10 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:10.401 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:10 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:10.401 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:10 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:10.401 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:10 smithi016 ceph-mon[29101]: Saving service rgw.foo.a spec with placement smithi016=foo.a;count:1 2024-03-26T04:17:10.401 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:10 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:10.401 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:10 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:10.401 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:10 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:17:10.401 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:10 smithi016 ceph-mon[37215]: from='client.24358 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-26T04:17:10.401 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:10 smithi016 ceph-mon[37215]: osdmap e59: 8 total, 8 up, 8 in 2024-03-26T04:17:10.401 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:10 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:10.401 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:10 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:10.402 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:10 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:10.402 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:10 smithi016 ceph-mon[37215]: Saving service rgw.foo.a spec with placement smithi016=foo.a;count:1 2024-03-26T04:17:10.402 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:10 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:10.402 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:10 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:10.402 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:10 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:17:11.350 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:11 smithi183 ceph-mon[33745]: pgmap v182: 4 pgs: 3 unknown, 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-26T04:17:11.350 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:11 smithi183 ceph-mon[33745]: osdmap e60: 8 total, 8 up, 8 in 2024-03-26T04:17:11.350 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:11 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/368235165' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T04:17:11.350 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:11 smithi183 ceph-mon[33745]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:17:11.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:11 smithi016 ceph-mon[29101]: pgmap v182: 4 pgs: 3 unknown, 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-26T04:17:11.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:11 smithi016 ceph-mon[29101]: osdmap e60: 8 total, 8 up, 8 in 2024-03-26T04:17:11.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:11 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/368235165' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T04:17:11.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:11 smithi016 ceph-mon[29101]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:17:11.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:11 smithi016 ceph-mon[37215]: pgmap v182: 4 pgs: 3 unknown, 1 active+clean; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-26T04:17:11.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:11 smithi016 ceph-mon[37215]: osdmap e60: 8 total, 8 up, 8 in 2024-03-26T04:17:11.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:11 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/368235165' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-26T04:17:11.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:11 smithi016 ceph-mon[37215]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:17:12.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:12 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/368235165' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T04:17:12.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:12 smithi016 ceph-mon[29101]: osdmap e61: 8 total, 8 up, 8 in 2024-03-26T04:17:12.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:12 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:12.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:12 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:12.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:12 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:17:12.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:12 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:17:12.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:12 smithi016 ceph-mon[29101]: Checking dashboard <-> RGW credentials 2024-03-26T04:17:12.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:12 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.183:0/4027334660' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T04:17:12.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:12 smithi016 ceph-mon[29101]: from='client.24388 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T04:17:12.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:12 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/368235165' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T04:17:12.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:12 smithi016 ceph-mon[37215]: osdmap e61: 8 total, 8 up, 8 in 2024-03-26T04:17:12.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:12 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:12.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:12 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:12.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:12 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:17:12.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:12 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:17:12.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:12 smithi016 ceph-mon[37215]: Checking dashboard <-> RGW credentials 2024-03-26T04:17:12.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:12 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.183:0/4027334660' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T04:17:12.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:12 smithi016 ceph-mon[37215]: from='client.24388 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T04:17:12.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:12 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/368235165' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-26T04:17:12.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:12 smithi183 ceph-mon[33745]: osdmap e61: 8 total, 8 up, 8 in 2024-03-26T04:17:12.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:12 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:12.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:12 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:12.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:12 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:17:12.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:12 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:17:12.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:12 smithi183 ceph-mon[33745]: Checking dashboard <-> RGW credentials 2024-03-26T04:17:12.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:12 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.183:0/4027334660' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T04:17:12.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:12 smithi183 ceph-mon[33745]: from='client.24388 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-26T04:17:13.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:13 smithi016 ceph-mon[29101]: pgmap v185: 36 pgs: 1 creating+peering, 1 active+clean, 34 unknown; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:17:13.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:13 smithi016 ceph-mon[29101]: from='client.24388 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-26T04:17:13.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:13 smithi016 ceph-mon[29101]: osdmap e62: 8 total, 8 up, 8 in 2024-03-26T04:17:13.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:13 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1239102722' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T04:17:13.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:13 smithi016 ceph-mon[29101]: from='client.34388 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T04:17:13.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:13 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2294055171' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T04:17:13.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:13 smithi016 ceph-mon[29101]: from='client.24412 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T04:17:13.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:13 smithi016 ceph-mon[37215]: pgmap v185: 36 pgs: 1 creating+peering, 1 active+clean, 34 unknown; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:17:13.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:13 smithi016 ceph-mon[37215]: from='client.24388 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-26T04:17:13.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:13 smithi016 ceph-mon[37215]: osdmap e62: 8 total, 8 up, 8 in 2024-03-26T04:17:13.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:13 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1239102722' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T04:17:13.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:13 smithi016 ceph-mon[37215]: from='client.34388 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T04:17:13.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:13 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2294055171' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T04:17:13.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:13 smithi016 ceph-mon[37215]: from='client.24412 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T04:17:13.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:13 smithi183 ceph-mon[33745]: pgmap v185: 36 pgs: 1 creating+peering, 1 active+clean, 34 unknown; 577 KiB data, 225 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:17:13.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:13 smithi183 ceph-mon[33745]: from='client.24388 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-26T04:17:13.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:13 smithi183 ceph-mon[33745]: osdmap e62: 8 total, 8 up, 8 in 2024-03-26T04:17:13.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:13 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1239102722' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T04:17:13.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:13 smithi183 ceph-mon[33745]: from='client.34388 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T04:17:13.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:13 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2294055171' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T04:17:13.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:13 smithi183 ceph-mon[33745]: from='client.24412 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-26T04:17:14.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:14 smithi016 ceph-mon[29101]: from='client.34388 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T04:17:14.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:14 smithi016 ceph-mon[29101]: from='client.24412 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T04:17:14.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:14 smithi016 ceph-mon[29101]: osdmap e63: 8 total, 8 up, 8 in 2024-03-26T04:17:14.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:14 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:14.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:14 smithi016 ceph-mon[37215]: from='client.34388 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T04:17:14.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:14 smithi016 ceph-mon[37215]: from='client.24412 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T04:17:14.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:14 smithi016 ceph-mon[37215]: osdmap e63: 8 total, 8 up, 8 in 2024-03-26T04:17:14.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:14 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:14.479 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:14 smithi183 ceph-mon[33745]: from='client.34388 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T04:17:14.479 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:14 smithi183 ceph-mon[33745]: from='client.24412 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-26T04:17:14.479 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:14 smithi183 ceph-mon[33745]: osdmap e63: 8 total, 8 up, 8 in 2024-03-26T04:17:14.479 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:14 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:14.795 DEBUG:teuthology.orchestra.run.smithi183:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.183 --placement '1;smithi183=iscsi.a' 2024-03-26T04:17:15.032 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:17:15.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:15 smithi016 ceph-mon[29101]: pgmap v188: 68 pgs: 29 active+clean, 5 creating+activating, 14 creating+peering, 20 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-26T04:17:15.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:15 smithi016 ceph-mon[29101]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T04:17:15.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:15 smithi016 ceph-mon[29101]: Cluster is now healthy 2024-03-26T04:17:15.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:15 smithi016 ceph-mon[29101]: osdmap e64: 8 total, 8 up, 8 in 2024-03-26T04:17:15.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:15 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1239102722' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T04:17:15.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:15 smithi016 ceph-mon[29101]: from='client.34388 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T04:17:15.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:15 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2294055171' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T04:17:15.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:15 smithi016 ceph-mon[29101]: from='client.24412 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T04:17:15.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:15 smithi016 ceph-mon[37215]: pgmap v188: 68 pgs: 29 active+clean, 5 creating+activating, 14 creating+peering, 20 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-26T04:17:15.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:15 smithi016 ceph-mon[37215]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T04:17:15.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:15 smithi016 ceph-mon[37215]: Cluster is now healthy 2024-03-26T04:17:15.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:15 smithi016 ceph-mon[37215]: osdmap e64: 8 total, 8 up, 8 in 2024-03-26T04:17:15.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:15 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1239102722' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T04:17:15.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:15 smithi016 ceph-mon[37215]: from='client.34388 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T04:17:15.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:15 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2294055171' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T04:17:15.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:15 smithi016 ceph-mon[37215]: from='client.24412 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T04:17:15.433 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:15 smithi183 ceph-mon[33745]: pgmap v188: 68 pgs: 29 active+clean, 5 creating+activating, 14 creating+peering, 20 unknown; 578 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-26T04:17:15.433 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:15 smithi183 ceph-mon[33745]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-26T04:17:15.433 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:15 smithi183 ceph-mon[33745]: Cluster is now healthy 2024-03-26T04:17:15.433 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:15 smithi183 ceph-mon[33745]: osdmap e64: 8 total, 8 up, 8 in 2024-03-26T04:17:15.433 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:15 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1239102722' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T04:17:15.433 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:15 smithi183 ceph-mon[33745]: from='client.34388 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T04:17:15.434 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:15 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2294055171' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T04:17:15.434 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:15 smithi183 ceph-mon[33745]: from='client.24412 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-26T04:17:16.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:16 smithi016 ceph-mon[29101]: from='client.34388 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T04:17:16.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:16 smithi016 ceph-mon[29101]: from='client.24412 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T04:17:16.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:16 smithi016 ceph-mon[29101]: osdmap e65: 8 total, 8 up, 8 in 2024-03-26T04:17:16.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:16 smithi016 ceph-mon[37215]: from='client.34388 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T04:17:16.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:16 smithi016 ceph-mon[37215]: from='client.24412 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T04:17:16.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:16 smithi016 ceph-mon[37215]: osdmap e65: 8 total, 8 up, 8 in 2024-03-26T04:17:16.426 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:16 smithi183 ceph-mon[33745]: from='client.34388 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T04:17:16.426 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:16 smithi183 ceph-mon[33745]: from='client.24412 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-26T04:17:16.426 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:16 smithi183 ceph-mon[33745]: osdmap e65: 8 total, 8 up, 8 in 2024-03-26T04:17:16.651 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:16.650+0000 7f1601f1a700 1 -- 172.21.15.183:0/319188894 >> v1:172.21.15.183:6789/0 conn(0x7f15fc1051c0 legacy=0x7f15fc107610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:16.651 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:16.651+0000 7f1601f1a700 1 -- 172.21.15.183:0/319188894 shutdown_connections 2024-03-26T04:17:16.652 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:16.651+0000 7f1601f1a700 1 -- 172.21.15.183:0/319188894 >> 172.21.15.183:0/319188894 conn(0x7f15fc0fc370 msgr2=0x7f15fc0fe790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:17:16.652 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:16.651+0000 7f1601f1a700 1 -- 172.21.15.183:0/319188894 shutdown_connections 2024-03-26T04:17:16.652 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:16.651+0000 7f1601f1a700 1 -- 172.21.15.183:0/319188894 wait complete. 2024-03-26T04:17:16.652 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:16.651+0000 7f1601f1a700 1 Processor -- start 2024-03-26T04:17:16.654 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:16.651+0000 7f1601f1a700 1 -- start start 2024-03-26T04:17:16.654 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:16.652+0000 7f1601f1a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f15fc10dc30 con 0x7f15fc108f00 2024-03-26T04:17:16.654 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:16.652+0000 7f1601f1a700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f15fc06b2f0 con 0x7f15fc100d90 2024-03-26T04:17:16.654 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:16.652+0000 7f1601f1a700 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f15fc1a46c0 con 0x7f15fc1051c0 2024-03-26T04:17:16.654 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:16.652+0000 7f15fb7fe700 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f15fc100d90 0x7f15fc19a120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:35274/0 (socket says 172.21.15.183:35274) 2024-03-26T04:17:16.654 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:16.652+0000 7f15fb7fe700 1 -- 172.21.15.183:0/1150821645 learned_addr learned my addr 172.21.15.183:0/1150821645 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-26T04:17:16.654 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:16.652+0000 7f15f8ff9700 1 -- 172.21.15.183:0/1150821645 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3198305701 0 0) 0x7f15fc06b2f0 con 0x7f15fc100d90 2024-03-26T04:17:16.654 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:16.652+0000 7f15f8ff9700 1 -- 172.21.15.183:0/1150821645 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15e40035b0 con 0x7f15fc100d90 2024-03-26T04:17:16.654 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:16.652+0000 7f15f8ff9700 1 -- 172.21.15.183:0/1150821645 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4112190565 0 0) 0x7f15fc1a46c0 con 0x7f15fc1051c0 2024-03-26T04:17:16.655 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:16.652+0000 7f15f8ff9700 1 -- 172.21.15.183:0/1150821645 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15fc06b2f0 con 0x7f15fc1051c0 2024-03-26T04:17:16.655 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:16.652+0000 7f15f8ff9700 1 -- 172.21.15.183:0/1150821645 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3094857309 0 0) 0x7f15fc10dc30 con 0x7f15fc108f00 2024-03-26T04:17:16.655 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:16.652+0000 7f15f8ff9700 1 -- 172.21.15.183:0/1150821645 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15fc1a46c0 con 0x7f15fc108f00 2024-03-26T04:17:16.655 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:16.652+0000 7f15f8ff9700 1 -- 172.21.15.183:0/1150821645 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4271699427 0 0) 0x7f15e40035b0 con 0x7f15fc100d90 2024-03-26T04:17:16.655 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:16.652+0000 7f15f8ff9700 1 -- 172.21.15.183:0/1150821645 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15fc10dc30 con 0x7f15fc100d90 2024-03-26T04:17:16.655 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:16.652+0000 7f15f8ff9700 1 -- 172.21.15.183:0/1150821645 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f15ec001930 con 0x7f15fc100d90 2024-03-26T04:17:16.655 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:16.652+0000 7f15f8ff9700 1 -- 172.21.15.183:0/1150821645 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4109311255 0 0) 0x7f15fc10dc30 con 0x7f15fc100d90 2024-03-26T04:17:16.655 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:16.652+0000 7f15f8ff9700 1 -- 172.21.15.183:0/1150821645 >> v1:172.21.15.16:6790/0 conn(0x7f15fc1051c0 legacy=0x7f15fc19d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:16.655 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:16.652+0000 7f15f8ff9700 1 -- 172.21.15.183:0/1150821645 >> v1:172.21.15.16:6789/0 conn(0x7f15fc108f00 legacy=0x7f15fc1a0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:16.655 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:16.652+0000 7f15f8ff9700 1 -- 172.21.15.183:0/1150821645 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15fc1a58c0 con 0x7f15fc100d90 2024-03-26T04:17:16.655 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:16.652+0000 7f1601f1a700 1 -- 172.21.15.183:0/1150821645 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15fc1a48b0 con 0x7f15fc100d90 2024-03-26T04:17:16.655 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:16.652+0000 7f1601f1a700 1 -- 172.21.15.183:0/1150821645 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f15fc1a4db0 con 0x7f15fc100d90 2024-03-26T04:17:16.655 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:16.653+0000 7f15f8ff9700 1 -- 172.21.15.183:0/1150821645 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f15ec001cc0 con 0x7f15fc100d90 2024-03-26T04:17:16.655 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:16.653+0000 7f15f8ff9700 1 -- 172.21.15.183:0/1150821645 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f15ec005d30 con 0x7f15fc100d90 2024-03-26T04:17:16.656 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:16.653+0000 7f1601f1a700 1 -- 172.21.15.183:0/1150821645 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15e8005150 con 0x7f15fc100d90 2024-03-26T04:17:16.656 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:16.654+0000 7f15f8ff9700 1 -- 172.21.15.183:0/1150821645 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 15) v1 ==== 91363+0+0 (unknown 160314185 0 0) 0x7f15ec003950 con 0x7f15fc100d90 2024-03-26T04:17:16.656 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:16.654+0000 7f15f8ff9700 1 -- 172.21.15.183:0/1150821645 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(66..66 src has 1..66) v4 ==== 5915+0+0 (unknown 800801731 0 0) 0x7f15ec088760 con 0x7f15fc100d90 2024-03-26T04:17:16.659 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:16.659+0000 7f15f8ff9700 1 -- 172.21.15.183:0/1150821645 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f15ec052000 con 0x7f15fc100d90 2024-03-26T04:17:16.867 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:16.866+0000 7f1601f1a700 1 -- 172.21.15.183:0/1150821645 --> v1:172.21.15.16:6800/4042489921 -- mgr_command(tid 0: {"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.183", "placement": "1;smithi183=iscsi.a", "target": ["mon-mgr", ""]}) v1 -- 0x7f15e8000c90 con 0x7f15e406e3f0 2024-03-26T04:17:16.875 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:16.874+0000 7f15f8ff9700 1 -- 172.21.15.183:0/1150821645 <== mgr.14152 v1:172.21.15.16:6800/4042489921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+35 (unknown 0 0 803663096) 0x7f15e8000c90 con 0x7f15e406e3f0 2024-03-26T04:17:16.875 INFO:teuthology.orchestra.run.smithi183.stdout:Scheduled iscsi.datapool update... 2024-03-26T04:17:16.877 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:16.876+0000 7f1601f1a700 1 -- 172.21.15.183:0/1150821645 >> v1:172.21.15.16:6800/4042489921 conn(0x7f15e406e3f0 legacy=0x7f15e40708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:16.877 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:16.876+0000 7f1601f1a700 1 -- 172.21.15.183:0/1150821645 >> v1:172.21.15.183:6789/0 conn(0x7f15fc100d90 legacy=0x7f15fc19a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:16.877 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:16.877+0000 7f1601f1a700 1 -- 172.21.15.183:0/1150821645 shutdown_connections 2024-03-26T04:17:16.877 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:16.877+0000 7f1601f1a700 1 -- 172.21.15.183:0/1150821645 >> 172.21.15.183:0/1150821645 conn(0x7f15fc0fc370 msgr2=0x7f15fc105600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:17:16.877 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:16.877+0000 7f1601f1a700 1 -- 172.21.15.183:0/1150821645 shutdown_connections 2024-03-26T04:17:16.877 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:16.877+0000 7f1601f1a700 1 -- 172.21.15.183:0/1150821645 wait complete. 2024-03-26T04:17:17.493 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:17 smithi183 ceph-mon[33745]: pgmap v191: 100 pgs: 50 active+clean, 5 creating+activating, 18 creating+peering, 27 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-03-26T04:17:17.494 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:17 smithi183 ceph-mon[33745]: osdmap e66: 8 total, 8 up, 8 in 2024-03-26T04:17:17.494 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:17 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1239102722' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T04:17:17.494 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:17 smithi183 ceph-mon[33745]: from='client.34388 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T04:17:17.494 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:17 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2294055171' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T04:17:17.494 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:17 smithi183 ceph-mon[33745]: from='client.24412 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T04:17:17.494 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:17 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:17.524 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-03-26T04:17:17.524 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-26T04:17:17.524 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-26T04:17:17.554 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:17 smithi016 ceph-mon[37215]: pgmap v191: 100 pgs: 50 active+clean, 5 creating+activating, 18 creating+peering, 27 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-03-26T04:17:17.555 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:17 smithi016 ceph-mon[37215]: osdmap e66: 8 total, 8 up, 8 in 2024-03-26T04:17:17.555 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:17 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1239102722' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T04:17:17.555 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:17 smithi016 ceph-mon[37215]: from='client.34388 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T04:17:17.555 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:17 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2294055171' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T04:17:17.555 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:17 smithi016 ceph-mon[37215]: from='client.24412 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T04:17:17.555 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:17 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:17.555 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:17 smithi016 ceph-mon[29101]: pgmap v191: 100 pgs: 50 active+clean, 5 creating+activating, 18 creating+peering, 27 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-03-26T04:17:17.555 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:17 smithi016 ceph-mon[29101]: osdmap e66: 8 total, 8 up, 8 in 2024-03-26T04:17:17.555 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:17 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1239102722' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T04:17:17.555 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:17 smithi016 ceph-mon[29101]: from='client.34388 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T04:17:17.555 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:17 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2294055171' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T04:17:17.556 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:17 smithi016 ceph-mon[29101]: from='client.24412 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-26T04:17:17.556 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:17 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:17.560 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-26T04:17:17.560 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-26T04:17:17.591 DEBUG:teuthology.orchestra.run.smithi183:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-d7d59826-eb26-11ee-95d0-87774f69a715@iscsi.iscsi.a.service 2024-03-26T04:17:17.634 INFO:tasks.cephadm:Adding prometheus.a on smithi183 2024-03-26T04:17:17.634 DEBUG:teuthology.orchestra.run.smithi183:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph orch apply prometheus '1;smithi183=a' 2024-03-26T04:17:17.908 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:17:18.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:18 smithi183 ceph-mon[33745]: from='client.24418 v1:172.21.15.183:0/1150821645' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.183", "placement": "1;smithi183=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:17:18.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:18 smithi183 ceph-mon[33745]: Saving service iscsi.datapool spec with placement smithi183=iscsi.a;count:1 2024-03-26T04:17:18.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:18 smithi183 ceph-mon[33745]: from='client.34388 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T04:17:18.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:18 smithi183 ceph-mon[33745]: from='client.24412 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T04:17:18.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:18 smithi183 ceph-mon[33745]: osdmap e67: 8 total, 8 up, 8 in 2024-03-26T04:17:18.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:18 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2294055171' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T04:17:18.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:18 smithi183 ceph-mon[33745]: from='client.24412 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T04:17:18.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:18 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1239102722' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T04:17:18.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:18 smithi183 ceph-mon[33745]: from='client.34388 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T04:17:18.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:18 smithi183 ceph-mon[33745]: from='client.24412 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T04:17:18.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:18 smithi183 ceph-mon[33745]: from='client.34388 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T04:17:18.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:18 smithi183 ceph-mon[33745]: osdmap e68: 8 total, 8 up, 8 in 2024-03-26T04:17:18.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:18 smithi016 ceph-mon[37215]: from='client.24418 v1:172.21.15.183:0/1150821645' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.183", "placement": "1;smithi183=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:17:18.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:18 smithi016 ceph-mon[37215]: Saving service iscsi.datapool spec with placement smithi183=iscsi.a;count:1 2024-03-26T04:17:18.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:18 smithi016 ceph-mon[37215]: from='client.34388 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T04:17:18.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:18 smithi016 ceph-mon[37215]: from='client.24412 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T04:17:18.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:18 smithi016 ceph-mon[37215]: osdmap e67: 8 total, 8 up, 8 in 2024-03-26T04:17:18.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:18 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2294055171' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T04:17:18.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:18 smithi016 ceph-mon[37215]: from='client.24412 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T04:17:18.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:18 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1239102722' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T04:17:18.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:18 smithi016 ceph-mon[37215]: from='client.34388 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T04:17:18.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:18 smithi016 ceph-mon[37215]: from='client.24412 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T04:17:18.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:18 smithi016 ceph-mon[37215]: from='client.34388 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T04:17:18.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:18 smithi016 ceph-mon[37215]: osdmap e68: 8 total, 8 up, 8 in 2024-03-26T04:17:18.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:18 smithi016 ceph-mon[29101]: from='client.24418 v1:172.21.15.183:0/1150821645' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.183", "placement": "1;smithi183=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:17:18.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:18 smithi016 ceph-mon[29101]: Saving service iscsi.datapool spec with placement smithi183=iscsi.a;count:1 2024-03-26T04:17:18.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:18 smithi016 ceph-mon[29101]: from='client.34388 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T04:17:18.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:18 smithi016 ceph-mon[29101]: from='client.24412 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-26T04:17:18.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:18 smithi016 ceph-mon[29101]: osdmap e67: 8 total, 8 up, 8 in 2024-03-26T04:17:18.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:18 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2294055171' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T04:17:18.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:18 smithi016 ceph-mon[29101]: from='client.24412 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T04:17:18.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:18 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1239102722' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T04:17:18.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:18 smithi016 ceph-mon[29101]: from='client.34388 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-26T04:17:18.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:18 smithi016 ceph-mon[29101]: from='client.24412 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T04:17:18.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:18 smithi016 ceph-mon[29101]: from='client.34388 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-26T04:17:18.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:18 smithi016 ceph-mon[29101]: osdmap e68: 8 total, 8 up, 8 in 2024-03-26T04:17:19.484 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:19.481+0000 7fafbfee8700 1 -- 172.21.15.183:0/2797654465 >> v1:172.21.15.183:6789/0 conn(0x7fafb8109f40 legacy=0x7fafb810c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:19.484 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:19.481+0000 7fafbfee8700 1 -- 172.21.15.183:0/2797654465 shutdown_connections 2024-03-26T04:17:19.484 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:19.481+0000 7fafbfee8700 1 -- 172.21.15.183:0/2797654465 >> 172.21.15.183:0/2797654465 conn(0x7fafb80fd430 msgr2=0x7fafb80ff870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:17:19.484 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:19.482+0000 7fafbfee8700 1 -- 172.21.15.183:0/2797654465 shutdown_connections 2024-03-26T04:17:19.484 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:19.482+0000 7fafbfee8700 1 -- 172.21.15.183:0/2797654465 wait complete. 2024-03-26T04:17:19.485 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:19.482+0000 7fafbfee8700 1 Processor -- start 2024-03-26T04:17:19.485 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:19.482+0000 7fafbfee8700 1 -- start start 2024-03-26T04:17:19.485 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:19.482+0000 7fafbfee8700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fafb8111150 con 0x7fafb81062b0 2024-03-26T04:17:19.485 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:19.482+0000 7fafbfee8700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fafb8197f40 con 0x7fafb8101e90 2024-03-26T04:17:19.485 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:19.482+0000 7fafbfee8700 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fafb81a46e0 con 0x7fafb8109f40 2024-03-26T04:17:19.485 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:19.482+0000 7fafbdc84700 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fafb8101e90 0x7fafb8197390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:51868/0 (socket says 172.21.15.183:51868) 2024-03-26T04:17:19.485 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:19.482+0000 7fafbdc84700 1 -- 172.21.15.183:0/2849739050 learned_addr learned my addr 172.21.15.183:0/2849739050 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-26T04:17:19.485 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:19.483+0000 7fafaeffd700 1 -- 172.21.15.183:0/2849739050 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 900514270 0 0) 0x7fafb8197f40 con 0x7fafb8101e90 2024-03-26T04:17:19.485 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:19.483+0000 7fafaeffd700 1 -- 172.21.15.183:0/2849739050 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafa40035b0 con 0x7fafb8101e90 2024-03-26T04:17:19.485 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:19.483+0000 7fafaeffd700 1 -- 172.21.15.183:0/2849739050 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3020945142 0 0) 0x7fafb81a46e0 con 0x7fafb8109f40 2024-03-26T04:17:19.485 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:19.483+0000 7fafaeffd700 1 -- 172.21.15.183:0/2849739050 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafb8197f40 con 0x7fafb8109f40 2024-03-26T04:17:19.485 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:19.483+0000 7fafaeffd700 1 -- 172.21.15.183:0/2849739050 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4024774143 0 0) 0x7fafb8111150 con 0x7fafb81062b0 2024-03-26T04:17:19.485 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:19.483+0000 7fafaeffd700 1 -- 172.21.15.183:0/2849739050 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafb81a46e0 con 0x7fafb81062b0 2024-03-26T04:17:19.486 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:19.483+0000 7fafaeffd700 1 -- 172.21.15.183:0/2849739050 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 379262081 0 0) 0x7fafa40035b0 con 0x7fafb8101e90 2024-03-26T04:17:19.486 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:19.483+0000 7fafaeffd700 1 -- 172.21.15.183:0/2849739050 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fafb8111150 con 0x7fafb8101e90 2024-03-26T04:17:19.486 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:19.483+0000 7fafaeffd700 1 -- 172.21.15.183:0/2849739050 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fafb4001930 con 0x7fafb8101e90 2024-03-26T04:17:19.486 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:19.483+0000 7fafaeffd700 1 -- 172.21.15.183:0/2849739050 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1477348042 0 0) 0x7fafb8197f40 con 0x7fafb8109f40 2024-03-26T04:17:19.486 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:19.483+0000 7fafaeffd700 1 -- 172.21.15.183:0/2849739050 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fafa40035b0 con 0x7fafb8109f40 2024-03-26T04:17:19.486 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:19.483+0000 7fafaeffd700 1 -- 172.21.15.183:0/2849739050 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fafb0004830 con 0x7fafb8109f40 2024-03-26T04:17:19.486 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:19.483+0000 7fafaeffd700 1 -- 172.21.15.183:0/2849739050 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1026981469 0 0) 0x7fafb8111150 con 0x7fafb8101e90 2024-03-26T04:17:19.486 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:19.483+0000 7fafaeffd700 1 -- 172.21.15.183:0/2849739050 >> v1:172.21.15.16:6790/0 conn(0x7fafb8109f40 legacy=0x7fafb81a0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:19.486 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:19.483+0000 7fafaeffd700 1 -- 172.21.15.183:0/2849739050 >> v1:172.21.15.16:6789/0 conn(0x7fafb81062b0 legacy=0x7fafb819d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:19.486 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:19.483+0000 7fafaeffd700 1 -- 172.21.15.183:0/2849739050 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb81a5880 con 0x7fafb8101e90 2024-03-26T04:17:19.486 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:19.483+0000 7fafbfee8700 1 -- 172.21.15.183:0/2849739050 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafb81a48d0 con 0x7fafb8101e90 2024-03-26T04:17:19.486 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:19.483+0000 7fafbfee8700 1 -- 172.21.15.183:0/2849739050 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fafb81a4dd0 con 0x7fafb8101e90 2024-03-26T04:17:19.486 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:19.483+0000 7fafaeffd700 1 -- 172.21.15.183:0/2849739050 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7fafb4003b30 con 0x7fafb8101e90 2024-03-26T04:17:19.486 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:19.483+0000 7fafaeffd700 1 -- 172.21.15.183:0/2849739050 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fafb4005c20 con 0x7fafb8101e90 2024-03-26T04:17:19.487 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:19.484+0000 7fafbfee8700 1 -- 172.21.15.183:0/2849739050 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf9c005150 con 0x7fafb8101e90 2024-03-26T04:17:19.487 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:19.485+0000 7fafaeffd700 1 -- 172.21.15.183:0/2849739050 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 15) v1 ==== 91363+0+0 (unknown 160314185 0 0) 0x7fafb40036a0 con 0x7fafb8101e90 2024-03-26T04:17:19.487 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:19.485+0000 7fafaeffd700 1 -- 172.21.15.183:0/2849739050 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(68..68 src has 1..68) v4 ==== 5942+0+0 (unknown 2675339414 0 0) 0x7fafb40886e0 con 0x7fafb8101e90 2024-03-26T04:17:19.490 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:19.490+0000 7fafaeffd700 1 -- 172.21.15.183:0/2849739050 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fafb4051f60 con 0x7fafb8101e90 2024-03-26T04:17:19.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:19 smithi183 ceph-mon[33745]: pgmap v194: 132 pgs: 80 active+clean, 5 creating+activating, 6 creating+peering, 41 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-26T04:17:19.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:19 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:19.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:19 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:19.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:19 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:19.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:19 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T04:17:19.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:19 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-26T04:17:19.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:19 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:17:19.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:19 smithi183 ceph-mon[33745]: Deploying daemon iscsi.iscsi.a on smithi183 2024-03-26T04:17:19.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:19 smithi016 ceph-mon[37215]: pgmap v194: 132 pgs: 80 active+clean, 5 creating+activating, 6 creating+peering, 41 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-26T04:17:19.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:19 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:19.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:19 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:19.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:19 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:19.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:19 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T04:17:19.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:19 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-26T04:17:19.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:19 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:17:19.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:19 smithi016 ceph-mon[37215]: Deploying daemon iscsi.iscsi.a on smithi183 2024-03-26T04:17:19.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:19 smithi016 ceph-mon[29101]: pgmap v194: 132 pgs: 80 active+clean, 5 creating+activating, 6 creating+peering, 41 unknown; 578 KiB data, 229 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-26T04:17:19.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:19 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:19.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:19 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:19.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:19 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:19.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:19 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-26T04:17:19.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:19 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-26T04:17:19.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:19 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:17:19.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:19 smithi016 ceph-mon[29101]: Deploying daemon iscsi.iscsi.a on smithi183 2024-03-26T04:17:19.704 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:19.704+0000 7fafbfee8700 1 -- 172.21.15.183:0/2849739050 --> v1:172.21.15.16:6800/4042489921 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi183=a", "target": ["mon-mgr", ""]}) v1 -- 0x7faf9c000bc0 con 0x7fafa406e670 2024-03-26T04:17:19.711 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:19.711+0000 7fafaeffd700 1 -- 172.21.15.183:0/2849739050 <== mgr.14152 v1:172.21.15.16:6800/4042489921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (unknown 0 0 1342662408) 0x7faf9c000bc0 con 0x7fafa406e670 2024-03-26T04:17:19.711 INFO:teuthology.orchestra.run.smithi183.stdout:Scheduled prometheus update... 2024-03-26T04:17:19.714 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:19.713+0000 7fafbfee8700 1 -- 172.21.15.183:0/2849739050 >> v1:172.21.15.16:6800/4042489921 conn(0x7fafa406e670 legacy=0x7fafa4070b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:19.714 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:19.713+0000 7fafbfee8700 1 -- 172.21.15.183:0/2849739050 >> v1:172.21.15.183:6789/0 conn(0x7fafb8101e90 legacy=0x7fafb8197390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:19.714 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:19.713+0000 7fafbfee8700 1 -- 172.21.15.183:0/2849739050 shutdown_connections 2024-03-26T04:17:19.714 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:19.713+0000 7fafbfee8700 1 -- 172.21.15.183:0/2849739050 >> 172.21.15.183:0/2849739050 conn(0x7fafb80fd430 msgr2=0x7fafb810c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:17:19.714 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:19.713+0000 7fafbfee8700 1 -- 172.21.15.183:0/2849739050 shutdown_connections 2024-03-26T04:17:19.714 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:19.713+0000 7fafbfee8700 1 -- 172.21.15.183:0/2849739050 wait complete. 2024-03-26T04:17:20.669 DEBUG:teuthology.orchestra.run.smithi183:prometheus.a> sudo journalctl -f -n 0 -u ceph-d7d59826-eb26-11ee-95d0-87774f69a715@prometheus.a.service 2024-03-26T04:17:20.671 INFO:tasks.cephadm:Adding node-exporter.a on smithi016 2024-03-26T04:17:20.671 INFO:tasks.cephadm:Adding node-exporter.b on smithi183 2024-03-26T04:17:20.671 DEBUG:teuthology.orchestra.run.smithi183:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph orch apply node-exporter '2;smithi016=a;smithi183=b' 2024-03-26T04:17:21.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:20 smithi183 ceph-mon[33745]: from='client.24439 v1:172.21.15.183:0/2849739050' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi183=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:17:21.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:20 smithi183 ceph-mon[33745]: Saving service prometheus spec with placement smithi183=a;count:1 2024-03-26T04:17:21.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:20 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:21.107 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:17:21.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:20 smithi016 ceph-mon[37215]: from='client.24439 v1:172.21.15.183:0/2849739050' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi183=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:17:21.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:20 smithi016 ceph-mon[37215]: Saving service prometheus spec with placement smithi183=a;count:1 2024-03-26T04:17:21.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:20 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:21.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:20 smithi016 ceph-mon[29101]: from='client.24439 v1:172.21.15.183:0/2849739050' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi183=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:17:21.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:20 smithi016 ceph-mon[29101]: Saving service prometheus spec with placement smithi183=a;count:1 2024-03-26T04:17:21.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:20 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:21.715 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:17:21 smithi183 systemd[1]: Starting Ceph iscsi.iscsi.a for d7d59826-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:17:22.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:21 smithi183 ceph-mon[33745]: pgmap v196: 132 pgs: 123 active+clean, 5 creating+peering, 4 unknown; 580 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 3.6 KiB/s wr, 168 op/s 2024-03-26T04:17:22.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:21 smithi016 ceph-mon[37215]: pgmap v196: 132 pgs: 123 active+clean, 5 creating+peering, 4 unknown; 580 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 3.6 KiB/s wr, 168 op/s 2024-03-26T04:17:22.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:21 smithi016 ceph-mon[29101]: pgmap v196: 132 pgs: 123 active+clean, 5 creating+peering, 4 unknown; 580 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 75 KiB/s rd, 3.6 KiB/s wr, 168 op/s 2024-03-26T04:17:22.875 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:22.873+0000 7fc65b935700 1 -- 172.21.15.183:0/1220977776 >> v1:172.21.15.183:6789/0 conn(0x7fc6541051f0 legacy=0x7fc654107640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:22.875 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:22.873+0000 7fc65b935700 1 -- 172.21.15.183:0/1220977776 shutdown_connections 2024-03-26T04:17:22.875 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:22.873+0000 7fc65b935700 1 -- 172.21.15.183:0/1220977776 >> 172.21.15.183:0/1220977776 conn(0x7fc6540fc380 msgr2=0x7fc6540fe7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:17:22.875 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:22.873+0000 7fc65b935700 1 -- 172.21.15.183:0/1220977776 shutdown_connections 2024-03-26T04:17:22.875 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:22.873+0000 7fc65b935700 1 -- 172.21.15.183:0/1220977776 wait complete. 2024-03-26T04:17:22.876 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:22.874+0000 7fc65b935700 1 Processor -- start 2024-03-26T04:17:22.876 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:22.874+0000 7fc65b935700 1 -- start start 2024-03-26T04:17:22.876 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:22.874+0000 7fc65b935700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc6541a44a0 con 0x7fc654108f30 2024-03-26T04:17:22.876 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:22.874+0000 7fc65b935700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc6541a56a0 con 0x7fc6541051f0 2024-03-26T04:17:22.876 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:22.874+0000 7fc65b935700 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc6541a68a0 con 0x7fc654100dc0 2024-03-26T04:17:22.876 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:22.874+0000 7fc658ed0700 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fc6541051f0 0x7fc65419f280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:51886/0 (socket says 172.21.15.183:51886) 2024-03-26T04:17:22.876 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:22.874+0000 7fc658ed0700 1 -- 172.21.15.183:0/2778464117 learned_addr learned my addr 172.21.15.183:0/2778464117 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-26T04:17:22.876 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:22.874+0000 7fc64a7fc700 1 -- 172.21.15.183:0/2778464117 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4209728307 0 0) 0x7fc6541a56a0 con 0x7fc6541051f0 2024-03-26T04:17:22.876 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:22.874+0000 7fc64a7fc700 1 -- 172.21.15.183:0/2778464117 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6400035b0 con 0x7fc6541051f0 2024-03-26T04:17:22.876 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:22.874+0000 7fc64a7fc700 1 -- 172.21.15.183:0/2778464117 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1727734145 0 0) 0x7fc6541a68a0 con 0x7fc654100dc0 2024-03-26T04:17:22.876 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:22.874+0000 7fc64a7fc700 1 -- 172.21.15.183:0/2778464117 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6541a56a0 con 0x7fc654100dc0 2024-03-26T04:17:22.876 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:22.874+0000 7fc64a7fc700 1 -- 172.21.15.183:0/2778464117 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1410466189 0 0) 0x7fc6541a44a0 con 0x7fc654108f30 2024-03-26T04:17:22.876 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:22.874+0000 7fc64a7fc700 1 -- 172.21.15.183:0/2778464117 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6541a68a0 con 0x7fc654108f30 2024-03-26T04:17:22.877 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:22.874+0000 7fc64a7fc700 1 -- 172.21.15.183:0/2778464117 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3313938402 0 0) 0x7fc6400035b0 con 0x7fc6541051f0 2024-03-26T04:17:22.877 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:22.874+0000 7fc64a7fc700 1 -- 172.21.15.183:0/2778464117 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6541a44a0 con 0x7fc6541051f0 2024-03-26T04:17:22.877 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:22.874+0000 7fc64a7fc700 1 -- 172.21.15.183:0/2778464117 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fc644001430 con 0x7fc6541051f0 2024-03-26T04:17:22.877 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:22.875+0000 7fc64a7fc700 1 -- 172.21.15.183:0/2778464117 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2126402369 0 0) 0x7fc6541a44a0 con 0x7fc6541051f0 2024-03-26T04:17:22.877 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:22.875+0000 7fc64a7fc700 1 -- 172.21.15.183:0/2778464117 >> v1:172.21.15.16:6790/0 conn(0x7fc654100dc0 legacy=0x7fc6541070c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:22.877 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:22.875+0000 7fc64a7fc700 1 -- 172.21.15.183:0/2778464117 >> v1:172.21.15.16:6789/0 conn(0x7fc654108f30 legacy=0x7fc6541a2ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:22.877 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:22.875+0000 7fc64a7fc700 1 -- 172.21.15.183:0/2778464117 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6541a7aa0 con 0x7fc6541051f0 2024-03-26T04:17:22.877 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:22.875+0000 7fc65b935700 1 -- 172.21.15.183:0/2778464117 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6541a6a90 con 0x7fc6541051f0 2024-03-26T04:17:22.877 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:22.875+0000 7fc65b935700 1 -- 172.21.15.183:0/2778464117 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6541a6f90 con 0x7fc6541051f0 2024-03-26T04:17:22.877 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:22.875+0000 7fc64a7fc700 1 -- 172.21.15.183:0/2778464117 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7fc644001b70 con 0x7fc6541051f0 2024-03-26T04:17:22.877 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:22.875+0000 7fc64a7fc700 1 -- 172.21.15.183:0/2778464117 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fc644004fa0 con 0x7fc6541051f0 2024-03-26T04:17:22.877 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:22.876+0000 7fc65b935700 1 -- 172.21.15.183:0/2778464117 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc638005150 con 0x7fc6541051f0 2024-03-26T04:17:22.878 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:22.876+0000 7fc64a7fc700 1 -- 172.21.15.183:0/2778464117 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 15) v1 ==== 91363+0+0 (unknown 160314185 0 0) 0x7fc644005100 con 0x7fc6541051f0 2024-03-26T04:17:22.878 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:22.877+0000 7fc64a7fc700 1 -- 172.21.15.183:0/2778464117 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(68..68 src has 1..68) v4 ==== 5942+0+0 (unknown 2675339414 0 0) 0x7fc644088a30 con 0x7fc6541051f0 2024-03-26T04:17:22.880 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:22.881+0000 7fc64a7fc700 1 -- 172.21.15.183:0/2778464117 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fc644052450 con 0x7fc6541051f0 2024-03-26T04:17:22.881 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:17:22 smithi183 podman[60465]: 2024-03-26 04:17:22.659901116 +0000 UTC m=+0.018096650 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T04:17:23.090 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:23.090+0000 7fc65b935700 1 -- 172.21.15.183:0/2778464117 --> v1:172.21.15.16:6800/4042489921 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi016=a;smithi183=b", "target": ["mon-mgr", ""]}) v1 -- 0x7fc638000bc0 con 0x7fc64006e440 2024-03-26T04:17:23.097 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:23.097+0000 7fc64a7fc700 1 -- 172.21.15.183:0/2778464117 <== mgr.14152 v1:172.21.15.16:6800/4042489921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (unknown 0 0 240551134) 0x7fc638000bc0 con 0x7fc64006e440 2024-03-26T04:17:23.097 INFO:teuthology.orchestra.run.smithi183.stdout:Scheduled node-exporter update... 2024-03-26T04:17:23.099 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:23.100+0000 7fc65b935700 1 -- 172.21.15.183:0/2778464117 >> v1:172.21.15.16:6800/4042489921 conn(0x7fc64006e440 legacy=0x7fc640070900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:23.099 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:23.100+0000 7fc65b935700 1 -- 172.21.15.183:0/2778464117 >> v1:172.21.15.183:6789/0 conn(0x7fc6541051f0 legacy=0x7fc65419f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:23.099 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:23.100+0000 7fc65b935700 1 -- 172.21.15.183:0/2778464117 shutdown_connections 2024-03-26T04:17:23.099 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:23.100+0000 7fc65b935700 1 -- 172.21.15.183:0/2778464117 >> 172.21.15.183:0/2778464117 conn(0x7fc6540fc380 msgr2=0x7fc65406aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:17:23.100 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:23.100+0000 7fc65b935700 1 -- 172.21.15.183:0/2778464117 shutdown_connections 2024-03-26T04:17:23.100 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:23.100+0000 7fc65b935700 1 -- 172.21.15.183:0/2778464117 wait complete. 2024-03-26T04:17:23.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:17:22 smithi183 podman[60465]: 2024-03-26 04:17:22.979806642 +0000 UTC m=+0.338002175 container create 77cff6840666b29ce345ea3aae672ce5d92c60e7517817ffd6ca3b6ad447cc2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0) 2024-03-26T04:17:23.658 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:17:23 smithi183 podman[60465]: 2024-03-26 04:17:23.400066274 +0000 UTC m=+0.758261807 container init 77cff6840666b29ce345ea3aae672ce5d92c60e7517817ffd6ca3b6ad447cc2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-26T04:17:23.658 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:17:23 smithi183 podman[60465]: 2024-03-26 04:17:23.403470407 +0000 UTC m=+0.761665947 container start 77cff6840666b29ce345ea3aae672ce5d92c60e7517817ffd6ca3b6ad447cc2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T04:17:23.658 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:17:23 smithi183 bash[60465]: 77cff6840666b29ce345ea3aae672ce5d92c60e7517817ffd6ca3b6ad447cc2e 2024-03-26T04:17:23.658 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:17:23 smithi183 systemd[1]: Started Ceph iscsi.iscsi.a for d7d59826-eb26-11ee-95d0-87774f69a715. 2024-03-26T04:17:24.029 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:23 smithi183 ceph-mon[33745]: pgmap v197: 132 pgs: 132 active+clean; 580 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 3.2 KiB/s wr, 149 op/s 2024-03-26T04:17:24.029 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:23 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:24.029 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:23 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:24.029 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:23 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:24.030 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:23 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:24.030 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:23 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:24.030 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:23 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:17:24.030 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:17:23 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug Started the configuration object watcher 2024-03-26T04:17:24.030 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:17:23 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug Checking for config object changes every 1s 2024-03-26T04:17:24.030 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:17:23 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug Processing osd blocklist entries for this node 2024-03-26T04:17:24.030 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:17:24 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug Reading the configuration object to update local LIO configuration 2024-03-26T04:17:24.030 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:17:24 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug Configuration does not have an entry for this host(smithi183.front.sepia.ceph.com) - nothing to define to LIO 2024-03-26T04:17:24.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:23 smithi016 ceph-mon[37215]: pgmap v197: 132 pgs: 132 active+clean; 580 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 3.2 KiB/s wr, 149 op/s 2024-03-26T04:17:24.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:23 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:24.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:23 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:24.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:23 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:24.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:23 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:24.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:23 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:24.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:23 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:17:24.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:23 smithi016 ceph-mon[29101]: pgmap v197: 132 pgs: 132 active+clean; 580 KiB data, 234 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 3.2 KiB/s wr, 149 op/s 2024-03-26T04:17:24.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:23 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:24.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:23 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:24.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:23 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:24.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:23 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:24.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:23 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:24.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:23 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:17:24.208 DEBUG:teuthology.orchestra.run.smithi016:node-exporter.a> sudo journalctl -f -n 0 -u ceph-d7d59826-eb26-11ee-95d0-87774f69a715@node-exporter.a.service 2024-03-26T04:17:24.210 DEBUG:teuthology.orchestra.run.smithi183:node-exporter.b> sudo journalctl -f -n 0 -u ceph-d7d59826-eb26-11ee-95d0-87774f69a715@node-exporter.b.service 2024-03-26T04:17:24.213 INFO:tasks.cephadm:Adding alertmanager.a on smithi016 2024-03-26T04:17:24.213 DEBUG:teuthology.orchestra.run.smithi183:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph orch apply alertmanager '1;smithi016=a' 2024-03-26T04:17:24.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:17:24 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-03-26T04:17:24.512 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:17:24.959 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:24 smithi183 ceph-mon[33745]: from='client.24445 v1:172.21.15.183:0/2778464117' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi016=a;smithi183=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:17:24.960 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:24 smithi183 ceph-mon[33745]: Saving service node-exporter spec with placement smithi016=a;smithi183=b;count:2 2024-03-26T04:17:24.960 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:24 smithi183 ceph-mon[33745]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-26T04:17:24.960 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:24 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:24.960 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:24 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.183:0/105411625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T04:17:25.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:24 smithi016 ceph-mon[29101]: from='client.24445 v1:172.21.15.183:0/2778464117' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi016=a;smithi183=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:17:25.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:24 smithi016 ceph-mon[29101]: Saving service node-exporter spec with placement smithi016=a;smithi183=b;count:2 2024-03-26T04:17:25.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:24 smithi016 ceph-mon[29101]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-26T04:17:25.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:24 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:25.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:24 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.183:0/105411625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T04:17:25.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:24 smithi016 ceph-mon[37215]: from='client.24445 v1:172.21.15.183:0/2778464117' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi016=a;smithi183=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:17:25.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:24 smithi016 ceph-mon[37215]: Saving service node-exporter spec with placement smithi016=a;smithi183=b;count:2 2024-03-26T04:17:25.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:24 smithi016 ceph-mon[37215]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-26T04:17:25.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:24 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:25.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:24 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.183:0/105411625' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-26T04:17:26.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:25 smithi183 ceph-mon[33745]: pgmap v198: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 4.7 KiB/s wr, 210 op/s 2024-03-26T04:17:26.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:25 smithi183 ceph-mon[33745]: osdmap e69: 8 total, 8 up, 8 in 2024-03-26T04:17:26.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:25 smithi016 ceph-mon[37215]: pgmap v198: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 4.7 KiB/s wr, 210 op/s 2024-03-26T04:17:26.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:25 smithi016 ceph-mon[37215]: osdmap e69: 8 total, 8 up, 8 in 2024-03-26T04:17:26.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:25 smithi016 ceph-mon[29101]: pgmap v198: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 94 KiB/s rd, 4.7 KiB/s wr, 210 op/s 2024-03-26T04:17:26.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:25 smithi016 ceph-mon[29101]: osdmap e69: 8 total, 8 up, 8 in 2024-03-26T04:17:26.783 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:26.782+0000 7fb1eb131700 1 -- 172.21.15.183:0/2112036773 >> v1:172.21.15.183:6789/0 conn(0x7fb1e41051a0 legacy=0x7fb1e41075f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:26.783 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:26.783+0000 7fb1eb131700 1 -- 172.21.15.183:0/2112036773 shutdown_connections 2024-03-26T04:17:26.784 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:26.783+0000 7fb1eb131700 1 -- 172.21.15.183:0/2112036773 >> 172.21.15.183:0/2112036773 conn(0x7fb1e40fc310 msgr2=0x7fb1e40fe750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:17:26.784 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:26.783+0000 7fb1eb131700 1 -- 172.21.15.183:0/2112036773 shutdown_connections 2024-03-26T04:17:26.784 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:26.783+0000 7fb1eb131700 1 -- 172.21.15.183:0/2112036773 wait complete. 2024-03-26T04:17:26.785 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:26.783+0000 7fb1eb131700 1 Processor -- start 2024-03-26T04:17:26.786 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:26.783+0000 7fb1eb131700 1 -- start start 2024-03-26T04:17:26.786 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:26.783+0000 7fb1eb131700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb1e410fff0 con 0x7fb1e4100d70 2024-03-26T04:17:26.786 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:26.783+0000 7fb1eb131700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb1e4197f10 con 0x7fb1e4108ee0 2024-03-26T04:17:26.786 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:26.783+0000 7fb1eb131700 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb1e41a46b0 con 0x7fb1e41051a0 2024-03-26T04:17:26.786 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:26.783+0000 7fb1e96ce700 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fb1e4108ee0 0x7fb1e41a0f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:51984/0 (socket says 172.21.15.183:51984) 2024-03-26T04:17:26.786 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:26.784+0000 7fb1e96ce700 1 -- 172.21.15.183:0/1754046010 learned_addr learned my addr 172.21.15.183:0/1754046010 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-26T04:17:26.786 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:26.784+0000 7fb1e1ffb700 1 -- 172.21.15.183:0/1754046010 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1151324064 0 0) 0x7fb1e4197f10 con 0x7fb1e4108ee0 2024-03-26T04:17:26.786 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:26.784+0000 7fb1e1ffb700 1 -- 172.21.15.183:0/1754046010 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1d40035b0 con 0x7fb1e4108ee0 2024-03-26T04:17:26.786 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:26.784+0000 7fb1e1ffb700 1 -- 172.21.15.183:0/1754046010 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 593410309 0 0) 0x7fb1e410fff0 con 0x7fb1e4100d70 2024-03-26T04:17:26.786 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:26.784+0000 7fb1e1ffb700 1 -- 172.21.15.183:0/1754046010 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1e4197f10 con 0x7fb1e4100d70 2024-03-26T04:17:26.786 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:26.784+0000 7fb1e1ffb700 1 -- 172.21.15.183:0/1754046010 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 975205352 0 0) 0x7fb1e41a46b0 con 0x7fb1e41051a0 2024-03-26T04:17:26.786 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:26.784+0000 7fb1e1ffb700 1 -- 172.21.15.183:0/1754046010 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1e410fff0 con 0x7fb1e41051a0 2024-03-26T04:17:26.786 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:26.784+0000 7fb1e1ffb700 1 -- 172.21.15.183:0/1754046010 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2278613320 0 0) 0x7fb1d40035b0 con 0x7fb1e4108ee0 2024-03-26T04:17:26.787 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:26.784+0000 7fb1e1ffb700 1 -- 172.21.15.183:0/1754046010 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1e41a46b0 con 0x7fb1e4108ee0 2024-03-26T04:17:26.787 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:26.784+0000 7fb1e1ffb700 1 -- 172.21.15.183:0/1754046010 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fb1dc001c90 con 0x7fb1e4108ee0 2024-03-26T04:17:26.787 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:26.784+0000 7fb1e1ffb700 1 -- 172.21.15.183:0/1754046010 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3239482181 0 0) 0x7fb1e4197f10 con 0x7fb1e4100d70 2024-03-26T04:17:26.787 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:26.784+0000 7fb1e1ffb700 1 -- 172.21.15.183:0/1754046010 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1d40035b0 con 0x7fb1e4100d70 2024-03-26T04:17:26.787 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:26.784+0000 7fb1e1ffb700 1 -- 172.21.15.183:0/1754046010 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fb1d80018b0 con 0x7fb1e4100d70 2024-03-26T04:17:26.787 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:26.784+0000 7fb1e1ffb700 1 -- 172.21.15.183:0/1754046010 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1351583321 0 0) 0x7fb1e410fff0 con 0x7fb1e41051a0 2024-03-26T04:17:26.787 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:26.784+0000 7fb1e1ffb700 1 -- 172.21.15.183:0/1754046010 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1e4197f10 con 0x7fb1e41051a0 2024-03-26T04:17:26.787 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:26.784+0000 7fb1e1ffb700 1 -- 172.21.15.183:0/1754046010 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fb1d0004340 con 0x7fb1e41051a0 2024-03-26T04:17:26.787 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:26.784+0000 7fb1e1ffb700 1 -- 172.21.15.183:0/1754046010 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1461540267 0 0) 0x7fb1e41a46b0 con 0x7fb1e4108ee0 2024-03-26T04:17:26.787 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:26.784+0000 7fb1e1ffb700 1 -- 172.21.15.183:0/1754046010 >> v1:172.21.15.16:6790/0 conn(0x7fb1e41051a0 legacy=0x7fb1e419d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:26.787 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:26.784+0000 7fb1e1ffb700 1 -- 172.21.15.183:0/1754046010 >> v1:172.21.15.16:6789/0 conn(0x7fb1e4100d70 legacy=0x7fb1e4197360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:26.787 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:26.784+0000 7fb1e1ffb700 1 -- 172.21.15.183:0/1754046010 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e41a5850 con 0x7fb1e4108ee0 2024-03-26T04:17:26.787 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:26.784+0000 7fb1eb131700 1 -- 172.21.15.183:0/1754046010 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1e41a48a0 con 0x7fb1e4108ee0 2024-03-26T04:17:26.787 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:26.785+0000 7fb1e1ffb700 1 -- 172.21.15.183:0/1754046010 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7fb1dc003eb0 con 0x7fb1e4108ee0 2024-03-26T04:17:26.788 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:26.785+0000 7fb1e1ffb700 1 -- 172.21.15.183:0/1754046010 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fb1dc006020 con 0x7fb1e4108ee0 2024-03-26T04:17:26.788 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:26.785+0000 7fb1eb131700 1 -- 172.21.15.183:0/1754046010 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1e41a4da0 con 0x7fb1e4108ee0 2024-03-26T04:17:26.788 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:26.785+0000 7fb1eb131700 1 -- 172.21.15.183:0/1754046010 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1e4068970 con 0x7fb1e4108ee0 2024-03-26T04:17:26.788 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:26.786+0000 7fb1e1ffb700 1 -- 172.21.15.183:0/1754046010 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 16) v1 ==== 91414+0+0 (unknown 2952574275 0 0) 0x7fb1dc01c6a0 con 0x7fb1e4108ee0 2024-03-26T04:17:26.788 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:26.787+0000 7fb1e1ffb700 1 -- 172.21.15.183:0/1754046010 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(69..69 src has 1..69) v4 ==== 5942+0+0 (unknown 2830370356 0 0) 0x7fb1dc088c40 con 0x7fb1e4108ee0 2024-03-26T04:17:26.791 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:26.791+0000 7fb1e1ffb700 1 -- 172.21.15.183:0/1754046010 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fb1dc0523c0 con 0x7fb1e4108ee0 2024-03-26T04:17:26.857 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:26 smithi183 ceph-mon[33745]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-26T04:17:27.062 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:27.061+0000 7fb1eb131700 1 -- 172.21.15.183:0/1754046010 --> v1:172.21.15.16:6800/4042489921 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi016=a", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1e41a5050 con 0x7fb1d406eae0 2024-03-26T04:17:27.069 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:27.068+0000 7fb1e1ffb700 1 -- 172.21.15.183:0/1754046010 <== mgr.14152 v1:172.21.15.16:6800/4042489921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (unknown 0 0 1850065467) 0x7fb1e41a5050 con 0x7fb1d406eae0 2024-03-26T04:17:27.069 INFO:teuthology.orchestra.run.smithi183.stdout:Scheduled alertmanager update... 2024-03-26T04:17:27.072 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:27.072+0000 7fb1eb131700 1 -- 172.21.15.183:0/1754046010 >> v1:172.21.15.16:6800/4042489921 conn(0x7fb1d406eae0 legacy=0x7fb1d4070fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:27.072 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:27.072+0000 7fb1eb131700 1 -- 172.21.15.183:0/1754046010 >> v1:172.21.15.183:6789/0 conn(0x7fb1e4108ee0 legacy=0x7fb1e41a0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:27.072 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:27.072+0000 7fb1eb131700 1 -- 172.21.15.183:0/1754046010 shutdown_connections 2024-03-26T04:17:27.073 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:27.072+0000 7fb1eb131700 1 -- 172.21.15.183:0/1754046010 >> 172.21.15.183:0/1754046010 conn(0x7fb1e40fc310 msgr2=0x7fb1e406aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:17:27.073 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:27.072+0000 7fb1eb131700 1 -- 172.21.15.183:0/1754046010 shutdown_connections 2024-03-26T04:17:27.073 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:27.072+0000 7fb1eb131700 1 -- 172.21.15.183:0/1754046010 wait complete. 2024-03-26T04:17:27.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:26 smithi016 ceph-mon[37215]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-26T04:17:27.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:26 smithi016 ceph-mon[29101]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-26T04:17:27.841 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:27 smithi183 ceph-mon[33745]: pgmap v200: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 4.4 KiB/s wr, 200 op/s 2024-03-26T04:17:27.841 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:27 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:28.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:27 smithi016 ceph-mon[37215]: pgmap v200: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 4.4 KiB/s wr, 200 op/s 2024-03-26T04:17:28.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:27 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:28.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:27 smithi016 ceph-mon[29101]: pgmap v200: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 90 KiB/s rd, 4.4 KiB/s wr, 200 op/s 2024-03-26T04:17:28.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:27 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:28.239 DEBUG:teuthology.orchestra.run.smithi016:alertmanager.a> sudo journalctl -f -n 0 -u ceph-d7d59826-eb26-11ee-95d0-87774f69a715@alertmanager.a.service 2024-03-26T04:17:28.242 INFO:tasks.cephadm:Adding grafana.a on smithi183 2024-03-26T04:17:28.242 DEBUG:teuthology.orchestra.run.smithi183:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph orch apply grafana '1;smithi183=a' 2024-03-26T04:17:28.525 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:17:28.913 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:28 smithi183 ceph-mon[33745]: from='client.24481 v1:172.21.15.183:0/1754046010' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi016=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:17:28.913 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:28 smithi183 ceph-mon[33745]: Saving service alertmanager spec with placement smithi016=a;count:1 2024-03-26T04:17:28.913 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:28 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:28.913 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:28 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:28.913 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:28 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:17:28.913 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:28 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:17:28.913 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:28 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:29.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:28 smithi016 ceph-mon[37215]: from='client.24481 v1:172.21.15.183:0/1754046010' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi016=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:17:29.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:28 smithi016 ceph-mon[37215]: Saving service alertmanager spec with placement smithi016=a;count:1 2024-03-26T04:17:29.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:28 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:29.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:28 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:29.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:28 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:17:29.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:28 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:17:29.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:28 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:29.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:28 smithi016 ceph-mon[29101]: from='client.24481 v1:172.21.15.183:0/1754046010' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi016=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:17:29.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:28 smithi016 ceph-mon[29101]: Saving service alertmanager spec with placement smithi016=a;count:1 2024-03-26T04:17:29.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:28 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:29.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:28 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:29.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:28 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:17:29.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:28 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:17:29.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:28 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:29.995 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:29 smithi183 ceph-mon[33745]: pgmap v201: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 3.6 KiB/s wr, 166 op/s 2024-03-26T04:17:29.995 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:29 smithi183 ceph-mon[33745]: Deploying daemon prometheus.a on smithi183 2024-03-26T04:17:29.995 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:29 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:30.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:29 smithi016 ceph-mon[37215]: pgmap v201: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 3.6 KiB/s wr, 166 op/s 2024-03-26T04:17:30.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:29 smithi016 ceph-mon[37215]: Deploying daemon prometheus.a on smithi183 2024-03-26T04:17:30.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:29 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:30.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:29 smithi016 ceph-mon[29101]: pgmap v201: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 3.6 KiB/s wr, 166 op/s 2024-03-26T04:17:30.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:29 smithi016 ceph-mon[29101]: Deploying daemon prometheus.a on smithi183 2024-03-26T04:17:30.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:29 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:30.235 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.232+0000 7fca0f760700 1 -- 172.21.15.183:0/3828751923 >> v1:172.21.15.183:6789/0 conn(0x7fca08100d80 legacy=0x7fca081011a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:30.235 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.233+0000 7fca0f760700 1 -- 172.21.15.183:0/3828751923 shutdown_connections 2024-03-26T04:17:30.235 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.233+0000 7fca0f760700 1 -- 172.21.15.183:0/3828751923 >> 172.21.15.183:0/3828751923 conn(0x7fca080fc360 msgr2=0x7fca080fe780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:17:30.235 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.233+0000 7fca0f760700 1 -- 172.21.15.183:0/3828751923 shutdown_connections 2024-03-26T04:17:30.236 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.233+0000 7fca0f760700 1 -- 172.21.15.183:0/3828751923 wait complete. 2024-03-26T04:17:30.236 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.233+0000 7fca0f760700 1 Processor -- start 2024-03-26T04:17:30.236 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.233+0000 7fca0f760700 1 -- start start 2024-03-26T04:17:30.236 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.233+0000 7fca0f760700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fca08110000 con 0x7fca081051b0 2024-03-26T04:17:30.236 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.233+0000 7fca0f760700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fca0806b310 con 0x7fca08100d80 2024-03-26T04:17:30.236 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.233+0000 7fca0f760700 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fca081a4690 con 0x7fca08108ef0 2024-03-26T04:17:30.236 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.233+0000 7fca0d4fc700 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fca08100d80 0x7fca0819a0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:50452/0 (socket says 172.21.15.183:50452) 2024-03-26T04:17:30.236 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.233+0000 7fca0d4fc700 1 -- 172.21.15.183:0/4020212666 learned_addr learned my addr 172.21.15.183:0/4020212666 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-26T04:17:30.236 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.234+0000 7fc9fe7fc700 1 -- 172.21.15.183:0/4020212666 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3148174287 0 0) 0x7fca0806b310 con 0x7fca08100d80 2024-03-26T04:17:30.236 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.234+0000 7fc9fe7fc700 1 -- 172.21.15.183:0/4020212666 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9f40035b0 con 0x7fca08100d80 2024-03-26T04:17:30.236 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.234+0000 7fc9fe7fc700 1 -- 172.21.15.183:0/4020212666 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2327692726 0 0) 0x7fca08110000 con 0x7fca081051b0 2024-03-26T04:17:30.236 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.234+0000 7fc9fe7fc700 1 -- 172.21.15.183:0/4020212666 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca0806b310 con 0x7fca081051b0 2024-03-26T04:17:30.236 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.234+0000 7fc9fe7fc700 1 -- 172.21.15.183:0/4020212666 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2126342471 0 0) 0x7fca081a4690 con 0x7fca08108ef0 2024-03-26T04:17:30.237 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.234+0000 7fc9fe7fc700 1 -- 172.21.15.183:0/4020212666 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca08110000 con 0x7fca08108ef0 2024-03-26T04:17:30.237 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.234+0000 7fc9fe7fc700 1 -- 172.21.15.183:0/4020212666 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1000749325 0 0) 0x7fca0806b310 con 0x7fca081051b0 2024-03-26T04:17:30.237 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.234+0000 7fc9fe7fc700 1 -- 172.21.15.183:0/4020212666 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca081a4690 con 0x7fca081051b0 2024-03-26T04:17:30.237 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.234+0000 7fc9fe7fc700 1 -- 172.21.15.183:0/4020212666 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fc9f80018b0 con 0x7fca081051b0 2024-03-26T04:17:30.237 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.234+0000 7fc9fe7fc700 1 -- 172.21.15.183:0/4020212666 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4208720905 0 0) 0x7fca08110000 con 0x7fca08108ef0 2024-03-26T04:17:30.237 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.234+0000 7fc9fe7fc700 1 -- 172.21.15.183:0/4020212666 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca0806b310 con 0x7fca08108ef0 2024-03-26T04:17:30.237 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.234+0000 7fc9fe7fc700 1 -- 172.21.15.183:0/4020212666 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 168047372 0 0) 0x7fc9f40035b0 con 0x7fca08100d80 2024-03-26T04:17:30.237 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.234+0000 7fc9fe7fc700 1 -- 172.21.15.183:0/4020212666 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca08110000 con 0x7fca08100d80 2024-03-26T04:17:30.237 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.234+0000 7fc9fe7fc700 1 -- 172.21.15.183:0/4020212666 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fca00001b70 con 0x7fca08108ef0 2024-03-26T04:17:30.237 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.234+0000 7fc9fe7fc700 1 -- 172.21.15.183:0/4020212666 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fca04001880 con 0x7fca08100d80 2024-03-26T04:17:30.237 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.234+0000 7fc9fe7fc700 1 -- 172.21.15.183:0/4020212666 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3165787428 0 0) 0x7fca081a4690 con 0x7fca081051b0 2024-03-26T04:17:30.237 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.234+0000 7fc9fe7fc700 1 -- 172.21.15.183:0/4020212666 >> v1:172.21.15.16:6790/0 conn(0x7fca08108ef0 legacy=0x7fca081a0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:30.237 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.234+0000 7fc9fe7fc700 1 -- 172.21.15.183:0/4020212666 >> v1:172.21.15.183:6789/0 conn(0x7fca08100d80 legacy=0x7fca0819a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:30.238 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.234+0000 7fc9fe7fc700 1 -- 172.21.15.183:0/4020212666 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca081a5890 con 0x7fca081051b0 2024-03-26T04:17:30.238 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.234+0000 7fca0f760700 1 -- 172.21.15.183:0/4020212666 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca081a4880 con 0x7fca081051b0 2024-03-26T04:17:30.238 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.234+0000 7fca0f760700 1 -- 172.21.15.183:0/4020212666 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fca081a4d80 con 0x7fca081051b0 2024-03-26T04:17:30.238 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.234+0000 7fc9fe7fc700 1 -- 172.21.15.183:0/4020212666 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7fc9f8003b30 con 0x7fca081051b0 2024-03-26T04:17:30.238 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.234+0000 7fc9fe7fc700 1 -- 172.21.15.183:0/4020212666 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fc9f8004af0 con 0x7fca081051b0 2024-03-26T04:17:30.238 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.235+0000 7fca0f760700 1 -- 172.21.15.183:0/4020212666 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca08068970 con 0x7fca081051b0 2024-03-26T04:17:30.239 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.236+0000 7fc9fe7fc700 1 -- 172.21.15.183:0/4020212666 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 16) v1 ==== 91414+0+0 (unknown 2952574275 0 0) 0x7fc9f80036a0 con 0x7fca081051b0 2024-03-26T04:17:30.239 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.236+0000 7fc9fe7fc700 1 -- 172.21.15.183:0/4020212666 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(69..69 src has 1..69) v4 ==== 5942+0+0 (unknown 2830370356 0 0) 0x7fc9f8087830 con 0x7fca081051b0 2024-03-26T04:17:30.241 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.241+0000 7fc9fe7fc700 1 -- 172.21.15.183:0/4020212666 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fc9f8050ef0 con 0x7fca081051b0 2024-03-26T04:17:30.461 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.461+0000 7fca0f760700 1 -- 172.21.15.183:0/4020212666 --> v1:172.21.15.16:6800/4042489921 -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi183=a", "target": ["mon-mgr", ""]}) v1 -- 0x7fca081a50a0 con 0x7fc9f406eb60 2024-03-26T04:17:30.467 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.467+0000 7fc9fe7fc700 1 -- 172.21.15.183:0/4020212666 <== mgr.14152 v1:172.21.15.16:6800/4042489921 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (unknown 0 0 664801700) 0x7fca081a50a0 con 0x7fc9f406eb60 2024-03-26T04:17:30.467 INFO:teuthology.orchestra.run.smithi183.stdout:Scheduled grafana update... 2024-03-26T04:17:30.470 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.469+0000 7fca0f760700 1 -- 172.21.15.183:0/4020212666 >> v1:172.21.15.16:6800/4042489921 conn(0x7fc9f406eb60 legacy=0x7fc9f4071020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:30.470 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.469+0000 7fca0f760700 1 -- 172.21.15.183:0/4020212666 >> v1:172.21.15.16:6789/0 conn(0x7fca081051b0 legacy=0x7fca0819d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:30.470 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.469+0000 7fca0f760700 1 -- 172.21.15.183:0/4020212666 shutdown_connections 2024-03-26T04:17:30.470 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.469+0000 7fca0f760700 1 -- 172.21.15.183:0/4020212666 >> 172.21.15.183:0/4020212666 conn(0x7fca080fc360 msgr2=0x7fca08107600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:17:30.470 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.469+0000 7fca0f760700 1 -- 172.21.15.183:0/4020212666 shutdown_connections 2024-03-26T04:17:30.470 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:30.469+0000 7fca0f760700 1 -- 172.21.15.183:0/4020212666 wait complete. 2024-03-26T04:17:31.705 DEBUG:teuthology.orchestra.run.smithi183:grafana.a> sudo journalctl -f -n 0 -u ceph-d7d59826-eb26-11ee-95d0-87774f69a715@grafana.a.service 2024-03-26T04:17:31.708 INFO:tasks.cephadm:Setting up client nodes... 2024-03-26T04:17:31.708 DEBUG:teuthology.orchestra.run.smithi016:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-26T04:17:31.731 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:31 smithi183 ceph-mon[33745]: pgmap v202: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 2.0 KiB/s wr, 83 op/s 2024-03-26T04:17:31.731 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:31 smithi183 ceph-mon[33745]: from='client.14592 v1:172.21.15.183:0/4020212666' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi183=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:17:31.731 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:31 smithi183 ceph-mon[33745]: Saving service grafana spec with placement smithi183=a;count:1 2024-03-26T04:17:31.731 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:31 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:31.738 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:31 smithi016 ceph-mon[29101]: pgmap v202: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 2.0 KiB/s wr, 83 op/s 2024-03-26T04:17:31.738 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:31 smithi016 ceph-mon[29101]: from='client.14592 v1:172.21.15.183:0/4020212666' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi183=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:17:31.738 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:31 smithi016 ceph-mon[29101]: Saving service grafana spec with placement smithi183=a;count:1 2024-03-26T04:17:31.738 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:31 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:31.738 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:31 smithi016 ceph-mon[37215]: pgmap v202: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 2.0 KiB/s wr, 83 op/s 2024-03-26T04:17:31.738 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:31 smithi016 ceph-mon[37215]: from='client.14592 v1:172.21.15.183:0/4020212666' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi183=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-26T04:17:31.739 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:31 smithi016 ceph-mon[37215]: Saving service grafana spec with placement smithi183=a;count:1 2024-03-26T04:17:31.739 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:31 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:31.962 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:17:33.610 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:33 smithi016 ceph-mon[37215]: pgmap v203: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 1.8 KiB/s wr, 72 op/s 2024-03-26T04:17:33.611 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:33 smithi016 ceph-mon[29101]: pgmap v203: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 1.8 KiB/s wr, 72 op/s 2024-03-26T04:17:33.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:33 smithi183 ceph-mon[33745]: pgmap v203: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 32 KiB/s rd, 1.8 KiB/s wr, 72 op/s 2024-03-26T04:17:33.792 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:17:33 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:17:33.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:33.851+0000 7f6e65b3e700 1 -- 172.21.15.16:0/2774307233 >> v1:172.21.15.16:6790/0 conn(0x7f6e601051a0 legacy=0x7f6e601075f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:33.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:33.851+0000 7f6e65b3e700 1 -- 172.21.15.16:0/2774307233 shutdown_connections 2024-03-26T04:17:33.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:33.851+0000 7f6e65b3e700 1 -- 172.21.15.16:0/2774307233 >> 172.21.15.16:0/2774307233 conn(0x7f6e600fc310 msgr2=0x7f6e600fe750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:17:33.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:33.851+0000 7f6e65b3e700 1 -- 172.21.15.16:0/2774307233 shutdown_connections 2024-03-26T04:17:33.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:33.851+0000 7f6e65b3e700 1 -- 172.21.15.16:0/2774307233 wait complete. 2024-03-26T04:17:33.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:33.852+0000 7f6e65b3e700 1 Processor -- start 2024-03-26T04:17:33.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:33.852+0000 7f6e65b3e700 1 -- start start 2024-03-26T04:17:33.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:33.852+0000 7f6e65b3e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6e6019a2b0 con 0x7f6e60108ee0 2024-03-26T04:17:33.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:33.852+0000 7f6e65b3e700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6e601a55c0 con 0x7f6e601051a0 2024-03-26T04:17:33.854 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:33.852+0000 7f6e65b3e700 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f6e601a6760 con 0x7f6e60100d70 2024-03-26T04:17:33.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:33.852+0000 7f6e5f7fe700 1 --1- >> v1:172.21.15.16:6790/0 conn(0x7f6e60100d70 0x7f6e60199760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6790/0 says I am v1:172.21.15.16:53094/0 (socket says 172.21.15.16:53094) 2024-03-26T04:17:33.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:33.852+0000 7f6e5f7fe700 1 -- 172.21.15.16:0/2941491812 learned_addr learned my addr 172.21.15.16:0/2941491812 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:17:33.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:33.852+0000 7f6e5cff9700 1 -- 172.21.15.16:0/2941491812 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4275937250 0 0) 0x7f6e601a6760 con 0x7f6e60100d70 2024-03-26T04:17:33.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:33.852+0000 7f6e5cff9700 1 -- 172.21.15.16:0/2941491812 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e480049e0 con 0x7f6e60100d70 2024-03-26T04:17:33.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:33.852+0000 7f6e5cff9700 1 -- 172.21.15.16:0/2941491812 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1564876865 0 0) 0x7f6e6019a2b0 con 0x7f6e60108ee0 2024-03-26T04:17:33.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:33.852+0000 7f6e5cff9700 1 -- 172.21.15.16:0/2941491812 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e601a6760 con 0x7f6e60108ee0 2024-03-26T04:17:33.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:33.853+0000 7f6e5cff9700 1 -- 172.21.15.16:0/2941491812 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3588985319 0 0) 0x7f6e601a55c0 con 0x7f6e601051a0 2024-03-26T04:17:33.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:33.853+0000 7f6e5cff9700 1 -- 172.21.15.16:0/2941491812 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e6019a2b0 con 0x7f6e601051a0 2024-03-26T04:17:33.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:33.853+0000 7f6e5cff9700 1 -- 172.21.15.16:0/2941491812 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 969969109 0 0) 0x7f6e480049e0 con 0x7f6e60100d70 2024-03-26T04:17:33.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:33.853+0000 7f6e5cff9700 1 -- 172.21.15.16:0/2941491812 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e601a55c0 con 0x7f6e60100d70 2024-03-26T04:17:33.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:33.853+0000 7f6e5cff9700 1 -- 172.21.15.16:0/2941491812 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f6e54001930 con 0x7f6e60100d70 2024-03-26T04:17:33.855 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:33.853+0000 7f6e5cff9700 1 -- 172.21.15.16:0/2941491812 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1306314370 0 0) 0x7f6e601a6760 con 0x7f6e60108ee0 2024-03-26T04:17:33.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:33.853+0000 7f6e5cff9700 1 -- 172.21.15.16:0/2941491812 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e480049e0 con 0x7f6e60108ee0 2024-03-26T04:17:33.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:33.853+0000 7f6e5cff9700 1 -- 172.21.15.16:0/2941491812 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f6e580013f0 con 0x7f6e60108ee0 2024-03-26T04:17:33.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:33.853+0000 7f6e5cff9700 1 -- 172.21.15.16:0/2941491812 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 991289483 0 0) 0x7f6e6019a2b0 con 0x7f6e601051a0 2024-03-26T04:17:33.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:33.853+0000 7f6e5cff9700 1 -- 172.21.15.16:0/2941491812 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e601a6760 con 0x7f6e601051a0 2024-03-26T04:17:33.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:33.853+0000 7f6e5cff9700 1 -- 172.21.15.16:0/2941491812 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f6e50001820 con 0x7f6e601051a0 2024-03-26T04:17:33.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:33.853+0000 7f6e5cff9700 1 -- 172.21.15.16:0/2941491812 <== mon.2 v1:172.21.15.16:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2769914224 0 0) 0x7f6e601a55c0 con 0x7f6e60100d70 2024-03-26T04:17:33.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:33.853+0000 7f6e5cff9700 1 -- 172.21.15.16:0/2941491812 >> v1:172.21.15.183:6789/0 conn(0x7f6e601051a0 legacy=0x7f6e6019f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:33.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:33.853+0000 7f6e5cff9700 1 -- 172.21.15.16:0/2941491812 >> v1:172.21.15.16:6789/0 conn(0x7f6e60108ee0 legacy=0x7f6e601a2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:33.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:33.853+0000 7f6e5cff9700 1 -- 172.21.15.16:0/2941491812 --> v1:172.21.15.16:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e601a7900 con 0x7f6e60100d70 2024-03-26T04:17:33.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:33.853+0000 7f6e65b3e700 1 -- 172.21.15.16:0/2941491812 --> v1:172.21.15.16:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e601a4610 con 0x7f6e60100d70 2024-03-26T04:17:33.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:33.853+0000 7f6e65b3e700 1 -- 172.21.15.16:0/2941491812 --> v1:172.21.15.16:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e601a4b10 con 0x7f6e60100d70 2024-03-26T04:17:33.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:33.853+0000 7f6e5cff9700 1 -- 172.21.15.16:0/2941491812 <== mon.2 v1:172.21.15.16:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f6e54001cc0 con 0x7f6e60100d70 2024-03-26T04:17:33.856 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:33.853+0000 7f6e5cff9700 1 -- 172.21.15.16:0/2941491812 <== mon.2 v1:172.21.15.16:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f6e54005c90 con 0x7f6e60100d70 2024-03-26T04:17:33.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:33.854+0000 7f6e65b3e700 1 -- 172.21.15.16:0/2941491812 --> v1:172.21.15.16:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e4c005150 con 0x7f6e60100d70 2024-03-26T04:17:33.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:33.855+0000 7f6e5cff9700 1 -- 172.21.15.16:0/2941491812 <== mon.2 v1:172.21.15.16:6790/0 7 ==== mgrmap(e 16) v1 ==== 91414+0+0 (unknown 2952574275 0 0) 0x7f6e5401c310 con 0x7f6e60100d70 2024-03-26T04:17:33.857 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:33.855+0000 7f6e5cff9700 1 -- 172.21.15.16:0/2941491812 <== mon.2 v1:172.21.15.16:6790/0 8 ==== osd_map(69..69 src has 1..69) v4 ==== 5942+0+0 (unknown 2830370356 0 0) 0x7f6e540888a0 con 0x7f6e60100d70 2024-03-26T04:17:33.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:33.860+0000 7f6e5cff9700 1 -- 172.21.15.16:0/2941491812 <== mon.2 v1:172.21.15.16:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f6e54051f60 con 0x7f6e60100d70 2024-03-26T04:17:34.145 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:34.145+0000 7f6e65b3e700 1 -- 172.21.15.16:0/2941491812 --> v1:172.21.15.16:6790/0 -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f6e4c005400 con 0x7f6e60100d70 2024-03-26T04:17:34.149 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:34.149+0000 7f6e5cff9700 1 -- 172.21.15.16:0/2941491812 <== mon.2 v1:172.21.15.16:6790/0 10 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v17) v1 ==== 170+0+59 (unknown 57899324 0 2199969286) 0x7f6e54053e90 con 0x7f6e60100d70 2024-03-26T04:17:34.149 INFO:teuthology.orchestra.run.smithi016.stdout:[client.0] 2024-03-26T04:17:34.149 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQBeTAJmK0bACBAAk6Mya2g+iz2GNLBXMgKKkQ== 2024-03-26T04:17:34.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:34.151+0000 7f6e65b3e700 1 -- 172.21.15.16:0/2941491812 >> v1:172.21.15.16:6800/4042489921 conn(0x7f6e4806ea90 legacy=0x7f6e48070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:34.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:34.151+0000 7f6e65b3e700 1 -- 172.21.15.16:0/2941491812 >> v1:172.21.15.16:6790/0 conn(0x7f6e60100d70 legacy=0x7f6e60199760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:34.151 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:34.151+0000 7f6e65b3e700 1 -- 172.21.15.16:0/2941491812 shutdown_connections 2024-03-26T04:17:34.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:34.151+0000 7f6e65b3e700 1 -- 172.21.15.16:0/2941491812 >> 172.21.15.16:0/2941491812 conn(0x7f6e600fc310 msgr2=0x7f6e6006a2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:17:34.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:34.151+0000 7f6e65b3e700 1 -- 172.21.15.16:0/2941491812 shutdown_connections 2024-03-26T04:17:34.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:34.151+0000 7f6e65b3e700 1 -- 172.21.15.16:0/2941491812 wait complete. 2024-03-26T04:17:34.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:34 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:17:34.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:34 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2941491812' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:17:34.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:34 smithi183 ceph-mon[33745]: from='client.34484 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:17:34.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:34 smithi183 ceph-mon[33745]: from='client.34484 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T04:17:34.848 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:34 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:17:34.848 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:34 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2941491812' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:17:34.848 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:34 smithi016 ceph-mon[37215]: from='client.34484 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:17:34.849 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:34 smithi016 ceph-mon[37215]: from='client.34484 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T04:17:34.849 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:34 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:17:34.849 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:34 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2941491812' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:17:34.849 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:34 smithi016 ceph-mon[29101]: from='client.34484 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:17:34.849 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:34 smithi016 ceph-mon[29101]: from='client.34484 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T04:17:34.881 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-26T04:17:34.881 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-26T04:17:34.881 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-26T04:17:34.969 DEBUG:teuthology.orchestra.run.smithi183:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-26T04:17:35.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:35 smithi183 ceph-mon[33745]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s 2024-03-26T04:17:35.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:35 smithi016 ceph-mon[29101]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s 2024-03-26T04:17:35.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:35 smithi016 ceph-mon[37215]: pgmap v204: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 102 B/s wr, 1 op/s 2024-03-26T04:17:35.959 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.b/config 2024-03-26T04:17:37.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:37 smithi183 ceph-mon[33745]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 991 B/s rd, 99 B/s wr, 1 op/s 2024-03-26T04:17:37.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:37 smithi016 ceph-mon[37215]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 991 B/s rd, 99 B/s wr, 1 op/s 2024-03-26T04:17:37.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:37 smithi016 ceph-mon[29101]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 991 B/s rd, 99 B/s wr, 1 op/s 2024-03-26T04:17:39.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:39 smithi183 ceph-mon[33745]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-03-26T04:17:39.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:39 smithi016 ceph-mon[37215]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-03-26T04:17:39.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:39 smithi016 ceph-mon[29101]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-03-26T04:17:40.586 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:40.586+0000 7f1046979700 1 -- 172.21.15.183:0/844629400 >> v1:172.21.15.183:6789/0 conn(0x7f1040108ec0 legacy=0x7f104010b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:40.586 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:40.586+0000 7f1046979700 1 -- 172.21.15.183:0/844629400 shutdown_connections 2024-03-26T04:17:40.587 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:40.586+0000 7f1046979700 1 -- 172.21.15.183:0/844629400 >> 172.21.15.183:0/844629400 conn(0x7f10400fbac0 msgr2=0x7f10400fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:17:40.587 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:40.586+0000 7f1046979700 1 -- 172.21.15.183:0/844629400 shutdown_connections 2024-03-26T04:17:40.587 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:40.586+0000 7f1046979700 1 -- 172.21.15.183:0/844629400 wait complete. 2024-03-26T04:17:40.588 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:40.586+0000 7f1046979700 1 Processor -- start 2024-03-26T04:17:40.588 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:40.586+0000 7f1046979700 1 -- start start 2024-03-26T04:17:40.588 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:40.587+0000 7f1046979700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1040110010 con 0x7f1040102310 2024-03-26T04:17:40.588 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:40.587+0000 7f1046979700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f1040100e60 con 0x7f1040105290 2024-03-26T04:17:40.589 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:40.587+0000 7f1046979700 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f10401a44c0 con 0x7f1040108ec0 2024-03-26T04:17:40.589 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:40.587+0000 7f103f7fe700 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f1040105290 0x7f104019d650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:48938/0 (socket says 172.21.15.183:48938) 2024-03-26T04:17:40.589 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:40.587+0000 7f103f7fe700 1 -- 172.21.15.183:0/3030500919 learned_addr learned my addr 172.21.15.183:0/3030500919 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-26T04:17:40.589 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:40.587+0000 7f103d7fa700 1 -- 172.21.15.183:0/3030500919 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3273444963 0 0) 0x7f1040100e60 con 0x7f1040105290 2024-03-26T04:17:40.589 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:40.587+0000 7f103d7fa700 1 -- 172.21.15.183:0/3030500919 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10280037f0 con 0x7f1040105290 2024-03-26T04:17:40.589 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:40.587+0000 7f103d7fa700 1 -- 172.21.15.183:0/3030500919 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3707102919 0 0) 0x7f1040110010 con 0x7f1040102310 2024-03-26T04:17:40.589 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:40.587+0000 7f103d7fa700 1 -- 172.21.15.183:0/3030500919 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1040100e60 con 0x7f1040102310 2024-03-26T04:17:40.589 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:40.587+0000 7f103d7fa700 1 -- 172.21.15.183:0/3030500919 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4107403214 0 0) 0x7f10401a44c0 con 0x7f1040108ec0 2024-03-26T04:17:40.589 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:40.587+0000 7f103d7fa700 1 -- 172.21.15.183:0/3030500919 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1040110010 con 0x7f1040108ec0 2024-03-26T04:17:40.589 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:40.587+0000 7f103d7fa700 1 -- 172.21.15.183:0/3030500919 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 250587000 0 0) 0x7f10280037f0 con 0x7f1040105290 2024-03-26T04:17:40.589 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:40.587+0000 7f103d7fa700 1 -- 172.21.15.183:0/3030500919 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f10401a44c0 con 0x7f1040105290 2024-03-26T04:17:40.589 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:40.587+0000 7f103d7fa700 1 -- 172.21.15.183:0/3030500919 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f10340018b0 con 0x7f1040105290 2024-03-26T04:17:40.589 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:40.587+0000 7f103d7fa700 1 -- 172.21.15.183:0/3030500919 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1422746679 0 0) 0x7f10401a44c0 con 0x7f1040105290 2024-03-26T04:17:40.589 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:40.588+0000 7f103d7fa700 1 -- 172.21.15.183:0/3030500919 >> v1:172.21.15.16:6790/0 conn(0x7f1040108ec0 legacy=0x7f10401a0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:40.590 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:40.588+0000 7f103d7fa700 1 -- 172.21.15.183:0/3030500919 >> v1:172.21.15.16:6789/0 conn(0x7f1040102310 legacy=0x7f1040199f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:40.590 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:40.588+0000 7f103d7fa700 1 -- 172.21.15.183:0/3030500919 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10401a56c0 con 0x7f1040105290 2024-03-26T04:17:40.590 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:40.588+0000 7f1046979700 1 -- 172.21.15.183:0/3030500919 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10401a46b0 con 0x7f1040105290 2024-03-26T04:17:40.590 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:40.588+0000 7f1046979700 1 -- 172.21.15.183:0/3030500919 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f10401a4bb0 con 0x7f1040105290 2024-03-26T04:17:40.591 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:40.588+0000 7f103d7fa700 1 -- 172.21.15.183:0/3030500919 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f1034001c60 con 0x7f1040105290 2024-03-26T04:17:40.591 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:40.588+0000 7f103d7fa700 1 -- 172.21.15.183:0/3030500919 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f1034005ce0 con 0x7f1040105290 2024-03-26T04:17:40.591 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:40.589+0000 7f1046979700 1 -- 172.21.15.183:0/3030500919 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f102c005150 con 0x7f1040105290 2024-03-26T04:17:40.591 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:40.589+0000 7f103d7fa700 1 -- 172.21.15.183:0/3030500919 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 16) v1 ==== 91414+0+0 (unknown 2952574275 0 0) 0x7f1034003950 con 0x7f1040105290 2024-03-26T04:17:40.591 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:40.590+0000 7f103d7fa700 1 -- 172.21.15.183:0/3030500919 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(69..69 src has 1..69) v4 ==== 5942+0+0 (unknown 2830370356 0 0) 0x7f1034088860 con 0x7f1040105290 2024-03-26T04:17:40.594 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:40.594+0000 7f103d7fa700 1 -- 172.21.15.183:0/3030500919 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f1034052000 con 0x7f1040105290 2024-03-26T04:17:40.874 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:40.873+0000 7f1046979700 1 -- 172.21.15.183:0/3030500919 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f102c005400 con 0x7f1040105290 2024-03-26T04:17:40.879 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:40.878+0000 7f103d7fa700 1 -- 172.21.15.183:0/3030500919 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v18) v1 ==== 170+0+59 (unknown 3508017507 0 2829107024) 0x7f1034053f30 con 0x7f1040105290 2024-03-26T04:17:40.879 INFO:teuthology.orchestra.run.smithi183.stdout:[client.1] 2024-03-26T04:17:40.879 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBkTAJmyf41NBAAj5KCmytT3OLrI6Rm2m37xg== 2024-03-26T04:17:40.881 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:40.880+0000 7f1046979700 1 -- 172.21.15.183:0/3030500919 >> v1:172.21.15.16:6800/4042489921 conn(0x7f102806e7b0 legacy=0x7f1028070c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:40.881 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:40.880+0000 7f1046979700 1 -- 172.21.15.183:0/3030500919 >> v1:172.21.15.183:6789/0 conn(0x7f1040105290 legacy=0x7f104019d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:40.881 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:40.881+0000 7f1046979700 1 -- 172.21.15.183:0/3030500919 shutdown_connections 2024-03-26T04:17:40.881 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:40.881+0000 7f1046979700 1 -- 172.21.15.183:0/3030500919 >> 172.21.15.183:0/3030500919 conn(0x7f10400fbac0 msgr2=0x7f10400fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:17:40.881 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:40.881+0000 7f1046979700 1 -- 172.21.15.183:0/3030500919 shutdown_connections 2024-03-26T04:17:40.881 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-26T04:17:40.881+0000 7f1046979700 1 -- 172.21.15.183:0/3030500919 wait complete. 2024-03-26T04:17:41.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:41 smithi183 ceph-mon[33745]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 1 op/s 2024-03-26T04:17:41.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:41 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.183:0/3030500919' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:17:41.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:41 smithi183 ceph-mon[33745]: from='client.24499 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:17:41.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:41 smithi183 ceph-mon[33745]: from='client.24499 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T04:17:41.792 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:17:41 smithi183 systemd[1]: Starting Ceph prometheus.a for d7d59826-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:17:41.844 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-26T04:17:41.844 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-26T04:17:41.844 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-26T04:17:41.890 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-26T04:17:41.891 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-26T04:17:41.891 DEBUG:teuthology.orchestra.run.smithi016:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph mgr dump --format=json 2024-03-26T04:17:41.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:41 smithi016 ceph-mon[37215]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 1 op/s 2024-03-26T04:17:41.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:41 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.183:0/3030500919' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:17:41.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:41 smithi016 ceph-mon[37215]: from='client.24499 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:17:41.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:41 smithi016 ceph-mon[37215]: from='client.24499 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T04:17:41.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:41 smithi016 ceph-mon[29101]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 1 op/s 2024-03-26T04:17:41.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:41 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.183:0/3030500919' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:17:41.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:41 smithi016 ceph-mon[29101]: from='client.24499 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-26T04:17:41.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:41 smithi016 ceph-mon[29101]: from='client.24499 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-26T04:17:42.111 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:17:42 smithi183 podman[61989]: 2024-03-26 04:17:41.955124298 +0000 UTC m=+0.018503295 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-26T04:17:42.112 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:17:42 smithi183 podman[61989]: 2024-03-26 04:17:42.112198157 +0000 UTC m=+0.175577194 container create d5ec4698a6dbd63cf3e79ca8096b9773a5c7affbf918ee1a135b3a8372297a71 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:17:42.143 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:17:42.364 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:17:42 smithi183 podman[61989]: 2024-03-26 04:17:42.339157982 +0000 UTC m=+0.402536973 container init d5ec4698a6dbd63cf3e79ca8096b9773a5c7affbf918ee1a135b3a8372297a71 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:17:42.364 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:17:42 smithi183 podman[61989]: 2024-03-26 04:17:42.341652758 +0000 UTC m=+0.405031754 container start d5ec4698a6dbd63cf3e79ca8096b9773a5c7affbf918ee1a135b3a8372297a71 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:17:42.792 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:17:42 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[62004]: ts=2024-03-26T04:17:42.364Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-26T04:17:42.792 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:17:42 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[62004]: ts=2024-03-26T04:17:42.364Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-26T04:17:42.792 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:17:42 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[62004]: ts=2024-03-26T04:17:42.364Z caller=main.go:570 level=info host_details="(Linux 5.14.0-432.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024 x86_64 smithi183 (none))" 2024-03-26T04:17:42.792 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:17:42 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[62004]: ts=2024-03-26T04:17:42.364Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-26T04:17:42.792 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:17:42 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[62004]: ts=2024-03-26T04:17:42.364Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-26T04:17:42.792 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:17:42 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[62004]: ts=2024-03-26T04:17:42.365Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-26T04:17:42.792 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:17:42 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[62004]: ts=2024-03-26T04:17:42.366Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-26T04:17:42.793 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:17:42 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[62004]: ts=2024-03-26T04:17:42.366Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-26T04:17:42.793 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:17:42 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[62004]: ts=2024-03-26T04:17:42.366Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-26T04:17:42.793 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:17:42 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[62004]: ts=2024-03-26T04:17:42.369Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-26T04:17:42.793 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:17:42 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[62004]: ts=2024-03-26T04:17:42.369Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.813µs 2024-03-26T04:17:42.793 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:17:42 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[62004]: ts=2024-03-26T04:17:42.369Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-26T04:17:42.793 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:17:42 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[62004]: ts=2024-03-26T04:17:42.369Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-03-26T04:17:42.793 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:17:42 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[62004]: ts=2024-03-26T04:17:42.369Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=22.518µs wal_replay_duration=244.368µs wbl_replay_duration=118ns total_replay_duration=285.937µs 2024-03-26T04:17:42.793 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:17:42 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[62004]: ts=2024-03-26T04:17:42.370Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-26T04:17:42.793 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:17:42 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[62004]: ts=2024-03-26T04:17:42.370Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-26T04:17:42.793 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:17:42 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[62004]: ts=2024-03-26T04:17:42.370Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-26T04:17:42.793 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:17:42 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[62004]: ts=2024-03-26T04:17:42.387Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=17.150828ms db_storage=746ns remote_storage=1.103µs web_handler=334ns query_engine=435ns scrape=306.041µs scrape_sd=59.388µs notify=695ns notify_sd=1.237µs rules=16.556374ms tracing=5.921µs 2024-03-26T04:17:42.793 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:17:42 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[62004]: ts=2024-03-26T04:17:42.387Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-26T04:17:42.794 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:17:42 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[62004]: ts=2024-03-26T04:17:42.387Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-26T04:17:42.794 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:17:42 smithi183 bash[61989]: d5ec4698a6dbd63cf3e79ca8096b9773a5c7affbf918ee1a135b3a8372297a71 2024-03-26T04:17:42.794 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:17:42 smithi183 systemd[1]: Started Ceph prometheus.a for d7d59826-eb26-11ee-95d0-87774f69a715. 2024-03-26T04:17:43.538 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:43 smithi016 ceph-mon[37215]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:17:43.539 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:43 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:43.539 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:43 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:43.539 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:43 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:43.539 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:43 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T04:17:43.539 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:43 smithi016 ceph-mon[29101]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:17:43.539 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:43 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:43.539 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:43 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:43.539 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:43 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:43.539 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:43 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T04:17:43.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:43 smithi183 ceph-mon[33745]: pgmap v208: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:17:43.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:43 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:43.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:43 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:43.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:43 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' 2024-03-26T04:17:43.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:43 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-26T04:17:43.792 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:17:43 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: ignoring --setuser ceph since I am not root 2024-03-26T04:17:43.792 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:17:43 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: ignoring --setgroup ceph since I am not root 2024-03-26T04:17:43.793 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:17:43 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ignoring --setuser ceph since I am not root 2024-03-26T04:17:43.793 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:17:43 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ignoring --setgroup ceph since I am not root 2024-03-26T04:17:43.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:43.898+0000 7fc21b7bb700 1 -- 172.21.15.16:0/1711129085 >> v1:172.21.15.16:6790/0 conn(0x7fc21410d790 legacy=0x7fc21410fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:43.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:43.898+0000 7fc21b7bb700 1 -- 172.21.15.16:0/1711129085 shutdown_connections 2024-03-26T04:17:43.899 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:43.898+0000 7fc21b7bb700 1 -- 172.21.15.16:0/1711129085 >> 172.21.15.16:0/1711129085 conn(0x7fc2140771b0 msgr2=0x7fc2140fe530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:17:43.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:43.899+0000 7fc21b7bb700 1 -- 172.21.15.16:0/1711129085 shutdown_connections 2024-03-26T04:17:43.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:43.899+0000 7fc21b7bb700 1 -- 172.21.15.16:0/1711129085 wait complete. 2024-03-26T04:17:43.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:43.899+0000 7fc21b7bb700 1 Processor -- start 2024-03-26T04:17:43.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:43.899+0000 7fc21b7bb700 1 -- start start 2024-03-26T04:17:43.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:43.899+0000 7fc21b7bb700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc2141a8e80 con 0x7fc214105d40 2024-03-26T04:17:43.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:43.899+0000 7fc21b7bb700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc2141aa060 con 0x7fc214109bc0 2024-03-26T04:17:43.902 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:43.899+0000 7fc21b7bb700 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc2141ab260 con 0x7fc21410d790 2024-03-26T04:17:43.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:43.899+0000 7fc219557700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc214105d40 0x7fc21410ced0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44840/0 (socket says 172.21.15.16:44840) 2024-03-26T04:17:43.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:43.899+0000 7fc219557700 1 -- 172.21.15.16:0/70145785 learned_addr learned my addr 172.21.15.16:0/70145785 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:17:43.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:43.900+0000 7fc2067fc700 1 -- 172.21.15.16:0/70145785 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2769536312 0 0) 0x7fc2141ab260 con 0x7fc21410d790 2024-03-26T04:17:43.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:43.900+0000 7fc2067fc700 1 -- 172.21.15.16:0/70145785 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2000035b0 con 0x7fc21410d790 2024-03-26T04:17:43.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:43.900+0000 7fc2067fc700 1 -- 172.21.15.16:0/70145785 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3488750330 0 0) 0x7fc2141a8e80 con 0x7fc214105d40 2024-03-26T04:17:43.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:43.900+0000 7fc2067fc700 1 -- 172.21.15.16:0/70145785 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2141ab260 con 0x7fc214105d40 2024-03-26T04:17:43.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:43.900+0000 7fc2067fc700 1 -- 172.21.15.16:0/70145785 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1094486284 0 0) 0x7fc2141aa060 con 0x7fc214109bc0 2024-03-26T04:17:43.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:43.900+0000 7fc2067fc700 1 -- 172.21.15.16:0/70145785 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2141a8e80 con 0x7fc214109bc0 2024-03-26T04:17:43.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:43.900+0000 7fc2067fc700 1 -- 172.21.15.16:0/70145785 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3755415817 0 0) 0x7fc2000035b0 con 0x7fc21410d790 2024-03-26T04:17:43.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:43.900+0000 7fc2067fc700 1 -- 172.21.15.16:0/70145785 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2141aa060 con 0x7fc21410d790 2024-03-26T04:17:43.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:43.900+0000 7fc2067fc700 1 -- 172.21.15.16:0/70145785 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fc20c001b80 con 0x7fc21410d790 2024-03-26T04:17:43.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:43.900+0000 7fc2067fc700 1 -- 172.21.15.16:0/70145785 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 169248574 0 0) 0x7fc2141ab260 con 0x7fc214105d40 2024-03-26T04:17:43.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:43.900+0000 7fc2067fc700 1 -- 172.21.15.16:0/70145785 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2000035b0 con 0x7fc214105d40 2024-03-26T04:17:43.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:43.900+0000 7fc2067fc700 1 -- 172.21.15.16:0/70145785 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fc2080012d0 con 0x7fc214105d40 2024-03-26T04:17:43.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:43.900+0000 7fc2067fc700 1 -- 172.21.15.16:0/70145785 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1954971006 0 0) 0x7fc2141a8e80 con 0x7fc214109bc0 2024-03-26T04:17:43.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:43.900+0000 7fc2067fc700 1 -- 172.21.15.16:0/70145785 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2141ab260 con 0x7fc214109bc0 2024-03-26T04:17:43.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:43.900+0000 7fc2067fc700 1 -- 172.21.15.16:0/70145785 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fc2100017f0 con 0x7fc214109bc0 2024-03-26T04:17:43.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:43.900+0000 7fc2067fc700 1 -- 172.21.15.16:0/70145785 <== mon.2 v1:172.21.15.16:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1235178775 0 0) 0x7fc2141aa060 con 0x7fc21410d790 2024-03-26T04:17:43.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:43.900+0000 7fc2067fc700 1 -- 172.21.15.16:0/70145785 >> v1:172.21.15.183:6789/0 conn(0x7fc214109bc0 legacy=0x7fc2141a3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:43.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:43.900+0000 7fc2067fc700 1 -- 172.21.15.16:0/70145785 >> v1:172.21.15.16:6789/0 conn(0x7fc214105d40 legacy=0x7fc21410ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:43.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:43.900+0000 7fc2067fc700 1 -- 172.21.15.16:0/70145785 --> v1:172.21.15.16:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2141ac460 con 0x7fc21410d790 2024-03-26T04:17:43.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:43.900+0000 7fc21b7bb700 1 -- 172.21.15.16:0/70145785 --> v1:172.21.15.16:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2141a9070 con 0x7fc21410d790 2024-03-26T04:17:43.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:43.900+0000 7fc21b7bb700 1 -- 172.21.15.16:0/70145785 --> v1:172.21.15.16:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2141a94f0 con 0x7fc21410d790 2024-03-26T04:17:43.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:43.901+0000 7fc2067fc700 1 -- 172.21.15.16:0/70145785 <== mon.2 v1:172.21.15.16:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7fc20c001e60 con 0x7fc21410d790 2024-03-26T04:17:43.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:43.901+0000 7fc2067fc700 1 -- 172.21.15.16:0/70145785 <== mon.2 v1:172.21.15.16:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fc20c004fc0 con 0x7fc21410d790 2024-03-26T04:17:43.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:43.901+0000 7fc21b7bb700 1 -- 172.21.15.16:0/70145785 --> v1:172.21.15.16:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1f8005120 con 0x7fc21410d790 2024-03-26T04:17:43.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:43.902+0000 7fc2067fc700 1 -- 172.21.15.16:0/70145785 <== mon.2 v1:172.21.15.16:6790/0 7 ==== mgrmap(e 17) v1 ==== 91428+0+0 (unknown 2925578053 0 0) 0x7fc20c003a90 con 0x7fc21410d790 2024-03-26T04:17:43.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:43.902+0000 7fc219557700 1 -- 172.21.15.16:0/70145785 >> v1:172.21.15.16:6800/4042489921 conn(0x7fc20006eb70 legacy=0x7fc200071030 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.16:6800/4042489921 2024-03-26T04:17:43.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:43.903+0000 7fc2067fc700 1 -- 172.21.15.16:0/70145785 <== mon.2 v1:172.21.15.16:6790/0 8 ==== osd_map(69..69 src has 1..69) v4 ==== 5942+0+0 (unknown 2830370356 0 0) 0x7fc20c087a80 con 0x7fc21410d790 2024-03-26T04:17:43.907 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:43.907+0000 7fc2067fc700 1 -- 172.21.15.16:0/70145785 <== mon.2 v1:172.21.15.16:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fc20c051200 con 0x7fc21410d790 2024-03-26T04:17:44.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:44.104+0000 7fc219557700 1 -- 172.21.15.16:0/70145785 >> v1:172.21.15.16:6800/4042489921 conn(0x7fc20006eb70 legacy=0x7fc200071030 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.16:6800/4042489921 2024-03-26T04:17:44.168 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:44.168+0000 7fc21b7bb700 1 -- 172.21.15.16:0/70145785 --> v1:172.21.15.16:6790/0 -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fc1f80053d0 con 0x7fc21410d790 2024-03-26T04:17:44.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:44.171+0000 7fc2067fc700 1 -- 172.21.15.16:0/70145785 <== mon.2 v1:172.21.15.16:6790/0 10 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+174803 (unknown 831476310 0 2640525395) 0x7fc20c053130 con 0x7fc21410d790 2024-03-26T04:17:44.176 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:17:44.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:44.174+0000 7fc21b7bb700 1 -- 172.21.15.16:0/70145785 >> v1:172.21.15.16:6800/4042489921 conn(0x7fc20006eb70 legacy=0x7fc200071030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T04:17:44.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:44.174+0000 7fc21b7bb700 1 -- 172.21.15.16:0/70145785 >> v1:172.21.15.16:6790/0 conn(0x7fc21410d790 legacy=0x7fc2141a7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:44.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:44.174+0000 7fc21b7bb700 1 -- 172.21.15.16:0/70145785 shutdown_connections 2024-03-26T04:17:44.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:44.174+0000 7fc21b7bb700 1 -- 172.21.15.16:0/70145785 >> 172.21.15.16:0/70145785 conn(0x7fc2140771b0 msgr2=0x7fc214100bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:17:44.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:44.174+0000 7fc21b7bb700 1 -- 172.21.15.16:0/70145785 shutdown_connections 2024-03-26T04:17:44.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:44.174+0000 7fc21b7bb700 1 -- 172.21.15.16:0/70145785 wait complete. 2024-03-26T04:17:44.412 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:17:44 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:17:44.104+0000 7f92c6f35200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T04:17:44.412 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:17:44 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:17:44.207+0000 7f92c6f35200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T04:17:44.503 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:17:44 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:17:44.071+0000 7fba82c8b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-26T04:17:44.503 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:17:44 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:17:44.165+0000 7fba82c8b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-26T04:17:44.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:44 smithi183 ceph-mon[33745]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T04:17:44.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:44 smithi183 ceph-mon[33745]: mgrmap e17: y(active, since 4m), standbys: x 2024-03-26T04:17:44.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:44 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/70145785' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T04:17:44.878 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:44 smithi016 ceph-mon[29101]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T04:17:44.879 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:44 smithi016 ceph-mon[29101]: mgrmap e17: y(active, since 4m), standbys: x 2024-03-26T04:17:44.879 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:44 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/70145785' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T04:17:44.879 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:44 smithi016 ceph-mon[37215]: from='mgr.14152 v1:172.21.15.16:0/477551245' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-26T04:17:44.879 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:44 smithi016 ceph-mon[37215]: mgrmap e17: y(active, since 4m), standbys: x 2024-03-26T04:17:44.879 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:44 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/70145785' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-26T04:17:44.966 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"y","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6800","nonce":4042489921}]},"active_addr":"172.21.15.16:6800/4042489921","active_change":"2024-03-26T04:12:43.759883+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.16:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":1783171530}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":3081221375}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":3077591423}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":2093082901}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":3716450293}]}]} 2024-03-26T04:17:44.970 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-26T04:17:44.970 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-26T04:17:44.970 DEBUG:teuthology.orchestra.run.smithi016:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T04:17:45.130 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:17:44 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:17:44.799+0000 7fba82c8b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T04:17:45.130 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:17:44 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:17:44.890+0000 7fba82c8b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T04:17:45.130 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:17:45 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:17:45.033+0000 7fba82c8b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T04:17:45.130 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:17:45 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:17:45.129+0000 7fba82c8b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T04:17:45.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:17:44 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:17:44.878+0000 7f92c6f35200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-26T04:17:45.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:17:44 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:17:44.975+0000 7f92c6f35200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-26T04:17:45.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:17:45 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:17:45.122+0000 7f92c6f35200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-26T04:17:45.269 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:17:45.479 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:17:45 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:17:45.215+0000 7fba82c8b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T04:17:45.512 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:17:45 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:17:45.242+0000 7f92c6f35200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-26T04:17:45.512 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:17:45 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:17:45.339+0000 7f92c6f35200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-26T04:17:45.770 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:17:45 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:17:45.479+0000 7fba82c8b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T04:17:45.771 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:17:45 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:17:45.588+0000 7fba82c8b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T04:17:45.771 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:17:45 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:17:45.674+0000 7fba82c8b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T04:17:45.771 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:17:45 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:17:45.769+0000 7fba82c8b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T04:17:45.818 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:17:45 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:17:45.612+0000 7f92c6f35200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-26T04:17:45.818 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:17:45 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:17:45.728+0000 7f92c6f35200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-26T04:17:45.818 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:17:45 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:17:45.818+0000 7f92c6f35200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-26T04:17:46.042 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:17:45 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:17:45.948+0000 7fba82c8b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T04:17:46.087 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:17:45 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:17:45.916+0000 7f92c6f35200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-26T04:17:46.413 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:17:46 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:17:46.100+0000 7f92c6f35200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-26T04:17:46.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:46.953+0000 7fccc96eb700 1 -- 172.21.15.16:0/1479865528 >> v1:172.21.15.16:6789/0 conn(0x7fccc4109c00 legacy=0x7fccc410bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:46.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:46.953+0000 7fccc96eb700 1 -- 172.21.15.16:0/1479865528 shutdown_connections 2024-03-26T04:17:46.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:46.953+0000 7fccc96eb700 1 -- 172.21.15.16:0/1479865528 >> 172.21.15.16:0/1479865528 conn(0x7fccc40fc820 msgr2=0x7fccc40fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:17:46.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:46.953+0000 7fccc96eb700 1 -- 172.21.15.16:0/1479865528 shutdown_connections 2024-03-26T04:17:46.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:46.953+0000 7fccc96eb700 1 -- 172.21.15.16:0/1479865528 wait complete. 2024-03-26T04:17:46.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:46.953+0000 7fccc96eb700 1 Processor -- start 2024-03-26T04:17:46.954 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:46.954+0000 7fccc96eb700 1 -- start start 2024-03-26T04:17:46.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:46.954+0000 7fccc96eb700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fccc41a6d30 con 0x7fccc4100f70 2024-03-26T04:17:46.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:46.954+0000 7fccc96eb700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fccc41a7f30 con 0x7fccc410d7a0 2024-03-26T04:17:46.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:46.954+0000 7fccc96eb700 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fccc41a9130 con 0x7fccc4109c00 2024-03-26T04:17:46.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:46.954+0000 7fccc2ffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fccc4100f70 0x7fccc406a320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59208/0 (socket says 172.21.15.16:59208) 2024-03-26T04:17:46.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:46.954+0000 7fccc2ffd700 1 -- 172.21.15.16:0/2367713684 learned_addr learned my addr 172.21.15.16:0/2367713684 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:17:46.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:46.954+0000 7fccabfff700 1 -- 172.21.15.16:0/2367713684 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 301507059 0 0) 0x7fccc41a6d30 con 0x7fccc4100f70 2024-03-26T04:17:46.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:46.954+0000 7fccabfff700 1 -- 172.21.15.16:0/2367713684 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccac0049e0 con 0x7fccc4100f70 2024-03-26T04:17:46.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:46.954+0000 7fccabfff700 1 -- 172.21.15.16:0/2367713684 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1184464596 0 0) 0x7fccc41a9130 con 0x7fccc4109c00 2024-03-26T04:17:46.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:46.954+0000 7fccabfff700 1 -- 172.21.15.16:0/2367713684 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccc41a6d30 con 0x7fccc4109c00 2024-03-26T04:17:46.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:46.954+0000 7fccabfff700 1 -- 172.21.15.16:0/2367713684 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1490207011 0 0) 0x7fccc41a7f30 con 0x7fccc410d7a0 2024-03-26T04:17:46.956 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:46.954+0000 7fccabfff700 1 -- 172.21.15.16:0/2367713684 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccc41a9130 con 0x7fccc410d7a0 2024-03-26T04:17:46.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:46.955+0000 7fccabfff700 1 -- 172.21.15.16:0/2367713684 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1461594471 0 0) 0x7fccc41a9130 con 0x7fccc410d7a0 2024-03-26T04:17:46.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:46.955+0000 7fccabfff700 1 -- 172.21.15.16:0/2367713684 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fccc41a7f30 con 0x7fccc410d7a0 2024-03-26T04:17:46.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:46.955+0000 7fccabfff700 1 -- 172.21.15.16:0/2367713684 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3051430872 0 0) 0x7fccac0049e0 con 0x7fccc4100f70 2024-03-26T04:17:46.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:46.955+0000 7fccabfff700 1 -- 172.21.15.16:0/2367713684 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fccc41a9130 con 0x7fccc4100f70 2024-03-26T04:17:46.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:46.955+0000 7fccabfff700 1 -- 172.21.15.16:0/2367713684 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2894539853 0 0) 0x7fccc41a6d30 con 0x7fccc4109c00 2024-03-26T04:17:46.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:46.955+0000 7fccabfff700 1 -- 172.21.15.16:0/2367713684 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fccac0049e0 con 0x7fccc4109c00 2024-03-26T04:17:46.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:46.955+0000 7fccabfff700 1 -- 172.21.15.16:0/2367713684 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fccbc001bf0 con 0x7fccc410d7a0 2024-03-26T04:17:46.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:46.955+0000 7fccabfff700 1 -- 172.21.15.16:0/2367713684 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fccb8001600 con 0x7fccc4100f70 2024-03-26T04:17:46.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:46.955+0000 7fccabfff700 1 -- 172.21.15.16:0/2367713684 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fccb4004510 con 0x7fccc4109c00 2024-03-26T04:17:46.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:46.955+0000 7fccabfff700 1 -- 172.21.15.16:0/2367713684 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3434338097 0 0) 0x7fccc41a7f30 con 0x7fccc410d7a0 2024-03-26T04:17:46.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:46.955+0000 7fccabfff700 1 -- 172.21.15.16:0/2367713684 >> v1:172.21.15.16:6790/0 conn(0x7fccc4109c00 legacy=0x7fccc41a1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:46.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:46.955+0000 7fccabfff700 1 -- 172.21.15.16:0/2367713684 >> v1:172.21.15.16:6789/0 conn(0x7fccc4100f70 legacy=0x7fccc406a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:46.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:46.955+0000 7fccabfff700 1 -- 172.21.15.16:0/2367713684 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc41aa330 con 0x7fccc410d7a0 2024-03-26T04:17:46.957 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:46.955+0000 7fccc96eb700 1 -- 172.21.15.16:0/2367713684 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccc41a9320 con 0x7fccc410d7a0 2024-03-26T04:17:46.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:46.955+0000 7fccc96eb700 1 -- 172.21.15.16:0/2367713684 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fccc41a97a0 con 0x7fccc410d7a0 2024-03-26T04:17:46.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:46.955+0000 7fccabfff700 1 -- 172.21.15.16:0/2367713684 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7fccbc003eb0 con 0x7fccc410d7a0 2024-03-26T04:17:46.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:46.955+0000 7fccabfff700 1 -- 172.21.15.16:0/2367713684 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fccbc004c90 con 0x7fccc410d7a0 2024-03-26T04:17:46.958 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:46.956+0000 7fccc96eb700 1 -- 172.21.15.16:0/2367713684 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccb0005120 con 0x7fccc410d7a0 2024-03-26T04:17:46.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:46.957+0000 7fccabfff700 1 -- 172.21.15.16:0/2367713684 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 17) v1 ==== 91428+0+0 (unknown 2925578053 0 0) 0x7fccbc003a00 con 0x7fccc410d7a0 2024-03-26T04:17:46.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:46.958+0000 7fccc2ffd700 1 -- 172.21.15.16:0/2367713684 >> v1:172.21.15.16:6800/4042489921 conn(0x7fccac06ebf0 legacy=0x7fccac0710b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.16:6800/4042489921 2024-03-26T04:17:46.959 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:46.958+0000 7fccabfff700 1 -- 172.21.15.16:0/2367713684 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(69..69 src has 1..69) v4 ==== 5942+0+0 (unknown 2830370356 0 0) 0x7fccbc087810 con 0x7fccc410d7a0 2024-03-26T04:17:46.961 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:46.961+0000 7fccabfff700 1 -- 172.21.15.16:0/2367713684 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fccbc050d00 con 0x7fccc410d7a0 2024-03-26T04:17:47.159 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:47.159+0000 7fccc2ffd700 1 -- 172.21.15.16:0/2367713684 >> v1:172.21.15.16:6800/4042489921 conn(0x7fccac06ebf0 legacy=0x7fccac0710b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.16:6800/4042489921 2024-03-26T04:17:47.174 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:47.173+0000 7fccc96eb700 1 -- 172.21.15.16:0/2367713684 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fccb00053d0 con 0x7fccc410d7a0 2024-03-26T04:17:47.175 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:47.175+0000 7fccabfff700 1 -- 172.21.15.16:0/2367713684 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+20555 (unknown 2171855796 0 1091242952) 0x7fccbc052e00 con 0x7fccc410d7a0 2024-03-26T04:17:47.176 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:17:47.176 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":69,"fsid":"d7d59826-eb26-11ee-95d0-87774f69a715","created":"2024-03-26T04:11:49.624863+0000","modified":"2024-03-26T04:17:25.482234+0000","last_up_change":"2024-03-26T04:16:57.936663+0000","last_in_change":"2024-03-26T04:16:40.057008+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T04:14:58.099362+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-26T04:17:09.012512+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":64,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-26T04:17:09.533803+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-26T04:17:11.811953+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-26T04:17:13.396838+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-26T04:17:15.480448+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e4e9d68-710d-4365-8810-e71718f37dee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6801","nonce":1132171550}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6802","nonce":1132171550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6804","nonce":1132171550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6803","nonce":1132171550}]},"public_addr":"172.21.15.16:6801/1132171550","cluster_addr":"172.21.15.16:6802/1132171550","heartbeat_back_addr":"172.21.15.16:6804/1132171550","heartbeat_front_addr":"172.21.15.16:6803/1132171550","state":["exists","up"]},{"osd":1,"uuid":"332e927f-79f4-4c33-b7f9-6d353d1721b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6805","nonce":317987982}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6806","nonce":317987982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6808","nonce":317987982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6807","nonce":317987982}]},"public_addr":"172.21.15.16:6805/317987982","cluster_addr":"172.21.15.16:6806/317987982","heartbeat_back_addr":"172.21.15.16:6808/317987982","heartbeat_front_addr":"172.21.15.16:6807/317987982","state":["exists","up"]},{"osd":2,"uuid":"b935affe-6e20-4226-87d9-12d4afe35679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6809","nonce":1107259766}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6810","nonce":1107259766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6812","nonce":1107259766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6811","nonce":1107259766}]},"public_addr":"172.21.15.16:6809/1107259766","cluster_addr":"172.21.15.16:6810/1107259766","heartbeat_back_addr":"172.21.15.16:6812/1107259766","heartbeat_front_addr":"172.21.15.16:6811/1107259766","state":["exists","up"]},{"osd":3,"uuid":"6c84a303-c00a-4a27-82c1-5e9ec1fb8d6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6813","nonce":1463789444}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6814","nonce":1463789444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6816","nonce":1463789444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6815","nonce":1463789444}]},"public_addr":"172.21.15.16:6813/1463789444","cluster_addr":"172.21.15.16:6814/1463789444","heartbeat_back_addr":"172.21.15.16:6816/1463789444","heartbeat_front_addr":"172.21.15.16:6815/1463789444","state":["exists","up"]},{"osd":4,"uuid":"750f2953-ae54-49a2-a428-9848615e9ddc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6800","nonce":879252532}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6801","nonce":879252532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6803","nonce":879252532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6802","nonce":879252532}]},"public_addr":"172.21.15.183:6800/879252532","cluster_addr":"172.21.15.183:6801/879252532","heartbeat_back_addr":"172.21.15.183:6803/879252532","heartbeat_front_addr":"172.21.15.183:6802/879252532","state":["exists","up"]},{"osd":5,"uuid":"0bbc822d-07fc-45c7-809d-a77e2cba6428","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6804","nonce":3356204259}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6805","nonce":3356204259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6807","nonce":3356204259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6806","nonce":3356204259}]},"public_addr":"172.21.15.183:6804/3356204259","cluster_addr":"172.21.15.183:6805/3356204259","heartbeat_back_addr":"172.21.15.183:6807/3356204259","heartbeat_front_addr":"172.21.15.183:6806/3356204259","state":["exists","up"]},{"osd":6,"uuid":"0de5f3f1-7a6e-4d11-9f99-db09de758531","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6808","nonce":777758685}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6809","nonce":777758685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6811","nonce":777758685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6810","nonce":777758685}]},"public_addr":"172.21.15.183:6808/777758685","cluster_addr":"172.21.15.183:6809/777758685","heartbeat_back_addr":"172.21.15.183:6811/777758685","heartbeat_front_addr":"172.21.15.183:6810/777758685","state":["exists","up"]},{"osd":7,"uuid":"6091e9e5-f241-4eed-81dd-970138c422c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6812","nonce":804876304}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6813","nonce":804876304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6815","nonce":804876304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6814","nonce":804876304}]},"public_addr":"172.21.15.183:6812/804876304","cluster_addr":"172.21.15.183:6813/804876304","heartbeat_back_addr":"172.21.15.183:6815/804876304","heartbeat_front_addr":"172.21.15.183:6814/804876304","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:14:05.613355+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:14:30.042974+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:14:55.072347+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:15:19.489278+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:15:42.363457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:16:07.018333+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:16:31.061483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:16:56.065175+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/285950115":"2024-03-27T04:12:43.759732+0000","172.21.15.16:0/292479134":"2024-03-27T04:12:43.759732+0000","172.21.15.16:6800/1828954920":"2024-03-27T04:12:43.759732+0000","172.21.15.16:0/1846690915":"2024-03-27T04:12:17.215822+0000","172.21.15.16:0/3092402688":"2024-03-27T04:12:17.215822+0000","172.21.15.16:0/1998037661":"2024-03-27T04:12:17.215822+0000","172.21.15.16:6800/3967690707":"2024-03-27T04:12:17.215822+0000","172.21.15.16:0/981571879":"2024-03-27T04:12:43.759732+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T04:17:47.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:47.178+0000 7fccc96eb700 1 -- 172.21.15.16:0/2367713684 >> v1:172.21.15.16:6800/4042489921 conn(0x7fccac06ebf0 legacy=0x7fccac0710b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T04:17:47.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:47.178+0000 7fccc96eb700 1 -- 172.21.15.16:0/2367713684 >> v1:172.21.15.183:6789/0 conn(0x7fccc410d7a0 legacy=0x7fccc41a53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:47.178 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:47.178+0000 7fccc96eb700 1 -- 172.21.15.16:0/2367713684 shutdown_connections 2024-03-26T04:17:47.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:47.178+0000 7fccc96eb700 1 -- 172.21.15.16:0/2367713684 >> 172.21.15.16:0/2367713684 conn(0x7fccc40fc820 msgr2=0x7fccc4110c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:17:47.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:47.178+0000 7fccc96eb700 1 -- 172.21.15.16:0/2367713684 shutdown_connections 2024-03-26T04:17:47.179 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:47.178+0000 7fccc96eb700 1 -- 172.21.15.16:0/2367713684 wait complete. 2024-03-26T04:17:47.227 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:17:46 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:17:46.907+0000 7fba82c8b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T04:17:47.227 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:17:47 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:17:47.038+0000 7fba82c8b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T04:17:47.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:47 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2367713684' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:17:47.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:47 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2367713684' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:17:47.413 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:17:47 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:17:47.113+0000 7f92c6f35200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-26T04:17:47.413 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:17:47 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:17:47.257+0000 7f92c6f35200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-26T04:17:47.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:47 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2367713684' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:17:47.541 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:17:47 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:17:47.375+0000 7fba82c8b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T04:17:47.542 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:17:47 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:17:47.464+0000 7fba82c8b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T04:17:47.694 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:17:47 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:17:47.599+0000 7f92c6f35200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-26T04:17:47.775 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-26T04:17:47.775 DEBUG:teuthology.orchestra.run.smithi016:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph osd dump --format=json 2024-03-26T04:17:48.019 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:17:48.041 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:17:47 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:17:47.662+0000 7fba82c8b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:17:48.041 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:17:47 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:17:47.846+0000 7fba82c8b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T04:17:48.112 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:17:47 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:17:47.693+0000 7f92c6f35200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-26T04:17:48.112 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:17:47 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:17:47.904+0000 7f92c6f35200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:17:48.112 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:17:48 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:17:48.112+0000 7f92c6f35200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-26T04:17:48.393 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:17:48 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:17:48.131+0000 7fba82c8b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:17:48.662 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:17:48 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:17:48.411+0000 7f92c6f35200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-26T04:17:48.791 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:17:48 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:17:48.392+0000 7fba82c8b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T04:17:48.791 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:17:48 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:17:48.630+0000 7fba82c8b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T04:17:49.027 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:17:48 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:17:48.674+0000 7f92c6f35200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-26T04:17:49.027 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:17:48 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:17:48.920+0000 7f92c6f35200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-26T04:17:49.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.509+0000 7f04f436e700 1 -- 172.21.15.16:0/630893917 >> v1:172.21.15.16:6790/0 conn(0x7f04ec1020a0 legacy=0x7f04ec102480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:49.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.510+0000 7f04f436e700 1 -- 172.21.15.16:0/630893917 shutdown_connections 2024-03-26T04:17:49.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.510+0000 7f04f436e700 1 -- 172.21.15.16:0/630893917 >> 172.21.15.16:0/630893917 conn(0x7f04ec0fd950 msgr2=0x7f04ec0ffd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:17:49.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.510+0000 7f04f436e700 1 -- 172.21.15.16:0/630893917 shutdown_connections 2024-03-26T04:17:49.513 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.510+0000 7f04f436e700 1 -- 172.21.15.16:0/630893917 wait complete. 2024-03-26T04:17:49.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.510+0000 7f04f436e700 1 Processor -- start 2024-03-26T04:17:49.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.510+0000 7f04f436e700 1 -- start start 2024-03-26T04:17:49.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.510+0000 7f04f436e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f04ec1a8f40 con 0x7f04ec10e8d0 2024-03-26T04:17:49.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.510+0000 7f04f436e700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f04ec1aa0e0 con 0x7f04ec10ad30 2024-03-26T04:17:49.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.510+0000 7f04f436e700 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f04ec1ab2e0 con 0x7f04ec1020a0 2024-03-26T04:17:49.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.511+0000 7f04f210a700 1 --1- >> v1:172.21.15.16:6790/0 conn(0x7f04ec1020a0 0x7f04ec107270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6790/0 says I am v1:172.21.15.16:33244/0 (socket says 172.21.15.16:33244) 2024-03-26T04:17:49.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.511+0000 7f04f210a700 1 -- 172.21.15.16:0/3441991866 learned_addr learned my addr 172.21.15.16:0/3441991866 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:17:49.514 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.511+0000 7f04df7fe700 1 -- 172.21.15.16:0/3441991866 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2027165154 0 0) 0x7f04ec1ab2e0 con 0x7f04ec1020a0 2024-03-26T04:17:49.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.511+0000 7f04df7fe700 1 -- 172.21.15.16:0/3441991866 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04d80035b0 con 0x7f04ec1020a0 2024-03-26T04:17:49.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.511+0000 7f04df7fe700 1 -- 172.21.15.16:0/3441991866 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 202818502 0 0) 0x7f04ec1aa0e0 con 0x7f04ec10ad30 2024-03-26T04:17:49.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.511+0000 7f04df7fe700 1 -- 172.21.15.16:0/3441991866 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04ec1ab2e0 con 0x7f04ec10ad30 2024-03-26T04:17:49.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.511+0000 7f04df7fe700 1 -- 172.21.15.16:0/3441991866 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1500328953 0 0) 0x7f04ec1a8f40 con 0x7f04ec10e8d0 2024-03-26T04:17:49.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.511+0000 7f04df7fe700 1 -- 172.21.15.16:0/3441991866 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04ec1aa0e0 con 0x7f04ec10e8d0 2024-03-26T04:17:49.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.511+0000 7f04df7fe700 1 -- 172.21.15.16:0/3441991866 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2040096646 0 0) 0x7f04ec1ab2e0 con 0x7f04ec10ad30 2024-03-26T04:17:49.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.511+0000 7f04df7fe700 1 -- 172.21.15.16:0/3441991866 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04ec1a8f40 con 0x7f04ec10ad30 2024-03-26T04:17:49.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.511+0000 7f04df7fe700 1 -- 172.21.15.16:0/3441991866 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f04e0001870 con 0x7f04ec10ad30 2024-03-26T04:17:49.515 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.511+0000 7f04df7fe700 1 -- 172.21.15.16:0/3441991866 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3763273900 0 0) 0x7f04d80035b0 con 0x7f04ec1020a0 2024-03-26T04:17:49.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.511+0000 7f04df7fe700 1 -- 172.21.15.16:0/3441991866 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04ec1ab2e0 con 0x7f04ec1020a0 2024-03-26T04:17:49.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.511+0000 7f04df7fe700 1 -- 172.21.15.16:0/3441991866 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f04e8001440 con 0x7f04ec1020a0 2024-03-26T04:17:49.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.511+0000 7f04df7fe700 1 -- 172.21.15.16:0/3441991866 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 726841494 0 0) 0x7f04ec1a8f40 con 0x7f04ec10ad30 2024-03-26T04:17:49.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.511+0000 7f04df7fe700 1 -- 172.21.15.16:0/3441991866 >> v1:172.21.15.16:6790/0 conn(0x7f04ec1020a0 legacy=0x7f04ec107270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:49.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.511+0000 7f04df7fe700 1 -- 172.21.15.16:0/3441991866 >> v1:172.21.15.16:6789/0 conn(0x7f04ec10e8d0 legacy=0x7f04ec1a7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:49.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.511+0000 7f04df7fe700 1 -- 172.21.15.16:0/3441991866 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04ec1ac4e0 con 0x7f04ec10ad30 2024-03-26T04:17:49.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.511+0000 7f04f436e700 1 -- 172.21.15.16:0/3441991866 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04ec1aa2d0 con 0x7f04ec10ad30 2024-03-26T04:17:49.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.511+0000 7f04f436e700 1 -- 172.21.15.16:0/3441991866 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f04ec1aa750 con 0x7f04ec10ad30 2024-03-26T04:17:49.516 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.512+0000 7f04df7fe700 1 -- 172.21.15.16:0/3441991866 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f04e0003b30 con 0x7f04ec10ad30 2024-03-26T04:17:49.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.512+0000 7f04df7fe700 1 -- 172.21.15.16:0/3441991866 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f04e0004ab0 con 0x7f04ec10ad30 2024-03-26T04:17:49.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.512+0000 7f04f436e700 1 -- 172.21.15.16:0/3441991866 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04d0005120 con 0x7f04ec10ad30 2024-03-26T04:17:49.517 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.513+0000 7f04df7fe700 1 -- 172.21.15.16:0/3441991866 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 17) v1 ==== 91428+0+0 (unknown 2925578053 0 0) 0x7f04e00036a0 con 0x7f04ec10ad30 2024-03-26T04:17:49.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.514+0000 7f04f210a700 1 -- 172.21.15.16:0/3441991866 >> v1:172.21.15.16:6800/4042489921 conn(0x7f04d806e730 legacy=0x7f04d8070bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.16:6800/4042489921 2024-03-26T04:17:49.518 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.514+0000 7f04df7fe700 1 -- 172.21.15.16:0/3441991866 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(69..69 src has 1..69) v4 ==== 5942+0+0 (unknown 2830370356 0 0) 0x7f04e0087730 con 0x7f04ec10ad30 2024-03-26T04:17:49.519 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.518+0000 7f04df7fe700 1 -- 172.21.15.16:0/3441991866 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f04e0050df0 con 0x7f04ec10ad30 2024-03-26T04:17:49.715 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.715+0000 7f04f210a700 1 -- 172.21.15.16:0/3441991866 >> v1:172.21.15.16:6800/4042489921 conn(0x7f04d806e730 legacy=0x7f04d8070bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.16:6800/4042489921 2024-03-26T04:17:49.725 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.725+0000 7f04f436e700 1 -- 172.21.15.16:0/3441991866 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04d00053d0 con 0x7f04ec10ad30 2024-03-26T04:17:49.727 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.727+0000 7f04df7fe700 1 -- 172.21.15.16:0/3441991866 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+20555 (unknown 2171855796 0 1091242952) 0x7f04e0052d20 con 0x7f04ec10ad30 2024-03-26T04:17:49.727 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:17:49.727 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":69,"fsid":"d7d59826-eb26-11ee-95d0-87774f69a715","created":"2024-03-26T04:11:49.624863+0000","modified":"2024-03-26T04:17:25.482234+0000","last_up_change":"2024-03-26T04:16:57.936663+0000","last_in_change":"2024-03-26T04:16:40.057008+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-26T04:14:58.099362+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-03-26T04:17:09.012512+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":64,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-03-26T04:17:09.533803+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-26T04:17:11.811953+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-26T04:17:13.396838+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-26T04:17:15.480448+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5e4e9d68-710d-4365-8810-e71718f37dee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6801","nonce":1132171550}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6802","nonce":1132171550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6804","nonce":1132171550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6803","nonce":1132171550}]},"public_addr":"172.21.15.16:6801/1132171550","cluster_addr":"172.21.15.16:6802/1132171550","heartbeat_back_addr":"172.21.15.16:6804/1132171550","heartbeat_front_addr":"172.21.15.16:6803/1132171550","state":["exists","up"]},{"osd":1,"uuid":"332e927f-79f4-4c33-b7f9-6d353d1721b0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6805","nonce":317987982}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6806","nonce":317987982}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6808","nonce":317987982}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6807","nonce":317987982}]},"public_addr":"172.21.15.16:6805/317987982","cluster_addr":"172.21.15.16:6806/317987982","heartbeat_back_addr":"172.21.15.16:6808/317987982","heartbeat_front_addr":"172.21.15.16:6807/317987982","state":["exists","up"]},{"osd":2,"uuid":"b935affe-6e20-4226-87d9-12d4afe35679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6809","nonce":1107259766}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6810","nonce":1107259766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6812","nonce":1107259766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6811","nonce":1107259766}]},"public_addr":"172.21.15.16:6809/1107259766","cluster_addr":"172.21.15.16:6810/1107259766","heartbeat_back_addr":"172.21.15.16:6812/1107259766","heartbeat_front_addr":"172.21.15.16:6811/1107259766","state":["exists","up"]},{"osd":3,"uuid":"6c84a303-c00a-4a27-82c1-5e9ec1fb8d6c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6813","nonce":1463789444}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6814","nonce":1463789444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6816","nonce":1463789444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.16:6815","nonce":1463789444}]},"public_addr":"172.21.15.16:6813/1463789444","cluster_addr":"172.21.15.16:6814/1463789444","heartbeat_back_addr":"172.21.15.16:6816/1463789444","heartbeat_front_addr":"172.21.15.16:6815/1463789444","state":["exists","up"]},{"osd":4,"uuid":"750f2953-ae54-49a2-a428-9848615e9ddc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":30,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6800","nonce":879252532}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6801","nonce":879252532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6803","nonce":879252532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6802","nonce":879252532}]},"public_addr":"172.21.15.183:6800/879252532","cluster_addr":"172.21.15.183:6801/879252532","heartbeat_back_addr":"172.21.15.183:6803/879252532","heartbeat_front_addr":"172.21.15.183:6802/879252532","state":["exists","up"]},{"osd":5,"uuid":"0bbc822d-07fc-45c7-809d-a77e2cba6428","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6804","nonce":3356204259}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6805","nonce":3356204259}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6807","nonce":3356204259}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6806","nonce":3356204259}]},"public_addr":"172.21.15.183:6804/3356204259","cluster_addr":"172.21.15.183:6805/3356204259","heartbeat_back_addr":"172.21.15.183:6807/3356204259","heartbeat_front_addr":"172.21.15.183:6806/3356204259","state":["exists","up"]},{"osd":6,"uuid":"0de5f3f1-7a6e-4d11-9f99-db09de758531","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6808","nonce":777758685}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6809","nonce":777758685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6811","nonce":777758685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6810","nonce":777758685}]},"public_addr":"172.21.15.183:6808/777758685","cluster_addr":"172.21.15.183:6809/777758685","heartbeat_back_addr":"172.21.15.183:6811/777758685","heartbeat_front_addr":"172.21.15.183:6810/777758685","state":["exists","up"]},{"osd":7,"uuid":"6091e9e5-f241-4eed-81dd-970138c422c0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6812","nonce":804876304}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6813","nonce":804876304}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6815","nonce":804876304}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.183:6814","nonce":804876304}]},"public_addr":"172.21.15.183:6812/804876304","cluster_addr":"172.21.15.183:6813/804876304","heartbeat_back_addr":"172.21.15.183:6815/804876304","heartbeat_front_addr":"172.21.15.183:6814/804876304","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:14:05.613355+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:14:30.042974+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:14:55.072347+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:15:19.489278+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:15:42.363457+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:16:07.018333+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:16:31.061483+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-26T04:16:56.065175+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.16:0/285950115":"2024-03-27T04:12:43.759732+0000","172.21.15.16:0/292479134":"2024-03-27T04:12:43.759732+0000","172.21.15.16:6800/1828954920":"2024-03-27T04:12:43.759732+0000","172.21.15.16:0/1846690915":"2024-03-27T04:12:17.215822+0000","172.21.15.16:0/3092402688":"2024-03-27T04:12:17.215822+0000","172.21.15.16:0/1998037661":"2024-03-27T04:12:17.215822+0000","172.21.15.16:6800/3967690707":"2024-03-27T04:12:17.215822+0000","172.21.15.16:0/981571879":"2024-03-27T04:12:43.759732+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-26T04:17:49.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.729+0000 7f04f436e700 1 -- 172.21.15.16:0/3441991866 >> v1:172.21.15.16:6800/4042489921 conn(0x7f04d806e730 legacy=0x7f04d8070bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-26T04:17:49.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.729+0000 7f04f436e700 1 -- 172.21.15.16:0/3441991866 >> v1:172.21.15.183:6789/0 conn(0x7f04ec10ad30 legacy=0x7f04ec1a4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:49.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.729+0000 7f04f436e700 1 -- 172.21.15.16:0/3441991866 shutdown_connections 2024-03-26T04:17:49.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.729+0000 7f04f436e700 1 -- 172.21.15.16:0/3441991866 >> 172.21.15.16:0/3441991866 conn(0x7f04ec0fd950 msgr2=0x7f04ec107c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:17:49.729 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.729+0000 7f04f436e700 1 -- 172.21.15.16:0/3441991866 shutdown_connections 2024-03-26T04:17:49.730 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:49.729+0000 7f04f436e700 1 -- 172.21.15.16:0/3441991866 wait complete. 2024-03-26T04:17:49.779 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:17:49 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:17:49.433+0000 7fba82c8b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T04:17:49.976 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:49 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3441991866' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:17:49.976 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:17:49 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:17:49.781+0000 7f92c6f35200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-26T04:17:49.976 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:49 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3441991866' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:17:50.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:49 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3441991866' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-26T04:17:50.041 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:17:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:17:50.007+0000 7fba82c8b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T04:17:50.421 DEBUG:teuthology.orchestra.run.smithi016:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-26T04:17:50.422 DEBUG:teuthology.orchestra.run.smithi016:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-26T04:17:50.422 DEBUG:teuthology.orchestra.run.smithi016:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-26T04:17:50.423 DEBUG:teuthology.orchestra.run.smithi016:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-26T04:17:50.423 DEBUG:teuthology.orchestra.run.smithi016:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-26T04:17:50.423 DEBUG:teuthology.orchestra.run.smithi016:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-26T04:17:50.424 DEBUG:teuthology.orchestra.run.smithi016:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-26T04:17:50.425 DEBUG:teuthology.orchestra.run.smithi016:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-26T04:17:50.446 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:17:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:17:50.106+0000 7fba82c8b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T04:17:50.446 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:17:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:17:50.325+0000 7fba82c8b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T04:17:50.446 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:17:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: [26/Mar/2024:04:17:50] ENGINE Bus STARTING 2024-03-26T04:17:50.662 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:17:50 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:17:50.363+0000 7f92c6f35200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-26T04:17:50.662 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:17:50 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:17:50.495+0000 7f92c6f35200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-26T04:17:50.783 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:17:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: CherryPy Checker: 2024-03-26T04:17:50.783 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:17:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: The Application mounted at '' has an empty config. 2024-03-26T04:17:50.783 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:17:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: 2024-03-26T04:17:50.783 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:17:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: [26/Mar/2024:04:17:50] ENGINE Serving on http://:::9283 2024-03-26T04:17:50.783 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:17:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x[35589]: [26/Mar/2024:04:17:50] ENGINE Bus STARTED 2024-03-26T04:17:50.885 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:17:51.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:50 smithi183 ceph-mon[33745]: Standby manager daemon x restarted 2024-03-26T04:17:51.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:50 smithi183 ceph-mon[33745]: Standby manager daemon x started 2024-03-26T04:17:51.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:50 smithi183 ceph-mon[33745]: from='mgr.? v1:172.21.15.183:0/3672570139' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T04:17:51.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:50 smithi183 ceph-mon[33745]: from='mgr.? v1:172.21.15.183:0/3672570139' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:17:51.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:50 smithi183 ceph-mon[33745]: from='mgr.? v1:172.21.15.183:0/3672570139' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T04:17:51.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:50 smithi183 ceph-mon[33745]: from='mgr.? v1:172.21.15.183:0/3672570139' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:17:51.050 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:50 smithi016 ceph-mon[37215]: Standby manager daemon x restarted 2024-03-26T04:17:51.050 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:50 smithi016 ceph-mon[37215]: Standby manager daemon x started 2024-03-26T04:17:51.050 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:50 smithi016 ceph-mon[37215]: from='mgr.? v1:172.21.15.183:0/3672570139' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T04:17:51.050 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:50 smithi016 ceph-mon[37215]: from='mgr.? v1:172.21.15.183:0/3672570139' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:17:51.050 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:50 smithi016 ceph-mon[37215]: from='mgr.? v1:172.21.15.183:0/3672570139' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T04:17:51.050 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:50 smithi016 ceph-mon[37215]: from='mgr.? v1:172.21.15.183:0/3672570139' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:17:51.051 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:17:50 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:17:50.907+0000 7f92c6f35200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-26T04:17:51.051 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: [26/Mar/2024:04:17:51] ENGINE Bus STARTING 2024-03-26T04:17:51.051 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:50 smithi016 ceph-mon[29101]: Standby manager daemon x restarted 2024-03-26T04:17:51.051 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:50 smithi016 ceph-mon[29101]: Standby manager daemon x started 2024-03-26T04:17:51.051 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:50 smithi016 ceph-mon[29101]: from='mgr.? v1:172.21.15.183:0/3672570139' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-26T04:17:51.051 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:50 smithi016 ceph-mon[29101]: from='mgr.? v1:172.21.15.183:0/3672570139' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-26T04:17:51.051 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:50 smithi016 ceph-mon[29101]: from='mgr.? v1:172.21.15.183:0/3672570139' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-26T04:17:51.051 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:50 smithi016 ceph-mon[29101]: from='mgr.? v1:172.21.15.183:0/3672570139' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-26T04:17:51.226 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:17:51.227 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:17:51.230 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:17:51.230 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:17:51.234 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:17:51.241 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:17:51.348 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: CherryPy Checker: 2024-03-26T04:17:51.348 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: The Application mounted at '' has an empty config. 2024-03-26T04:17:51.348 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:17:51.348 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: [26/Mar/2024:04:17:51] ENGINE Serving on http://:::9283 2024-03-26T04:17:51.348 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: [26/Mar/2024:04:17:51] ENGINE Bus STARTED 2024-03-26T04:17:52.032 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[29101]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-26T04:17:52.032 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[29101]: Active manager daemon y restarted 2024-03-26T04:17:52.032 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[29101]: Activating manager daemon y 2024-03-26T04:17:52.032 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[29101]: osdmap e70: 8 total, 8 up, 8 in 2024-03-26T04:17:52.032 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:17:52.033 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[29101]: mgrmap e19: y(active, starting, since 0.0164264s), standbys: x 2024-03-26T04:17:52.033 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:17:52.033 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:17:52.033 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T04:17:52.033 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T04:17:52.033 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T04:17:52.033 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T04:17:52.033 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T04:17:52.033 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:17:52.033 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:17:52.033 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:17:52.033 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:17:52.034 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:17:52.034 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:17:52.034 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:17:52.034 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:17:52.034 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[29101]: Manager daemon y is now available 2024-03-26T04:17:52.034 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:17:52.034 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:17:52.034 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:17:52.034 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:17:52.034 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:17:52.034 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:17:52.034 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:17:52.034 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:17:52.035 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[37215]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-26T04:17:52.035 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[37215]: Active manager daemon y restarted 2024-03-26T04:17:52.035 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[37215]: Activating manager daemon y 2024-03-26T04:17:52.035 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[37215]: osdmap e70: 8 total, 8 up, 8 in 2024-03-26T04:17:52.035 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:17:52.035 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[37215]: mgrmap e19: y(active, starting, since 0.0164264s), standbys: x 2024-03-26T04:17:52.035 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:17:52.035 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:17:52.035 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T04:17:52.035 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T04:17:52.035 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T04:17:52.035 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T04:17:52.035 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T04:17:52.036 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:17:52.036 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:17:52.036 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:17:52.036 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:17:52.036 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:17:52.036 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:17:52.036 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:17:52.036 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:17:52.036 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[37215]: Manager daemon y is now available 2024-03-26T04:17:52.036 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:17:52.036 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:17:52.037 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:17:52.037 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:17:52.037 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:17:52.037 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:17:52.037 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:17:52.037 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:51 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:17:52.085 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:17:51 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:17:52.085 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:51 smithi183 ceph-mon[33745]: mgrmap e18: y(active, since 5m), standbys: x 2024-03-26T04:17:52.085 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:51 smithi183 ceph-mon[33745]: Active manager daemon y restarted 2024-03-26T04:17:52.085 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:51 smithi183 ceph-mon[33745]: Activating manager daemon y 2024-03-26T04:17:52.086 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:51 smithi183 ceph-mon[33745]: osdmap e70: 8 total, 8 up, 8 in 2024-03-26T04:17:52.086 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:51 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-26T04:17:52.086 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:51 smithi183 ceph-mon[33745]: mgrmap e19: y(active, starting, since 0.0164264s), standbys: x 2024-03-26T04:17:52.086 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:51 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-26T04:17:52.086 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:51 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-26T04:17:52.086 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:51 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-26T04:17:52.086 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:51 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-26T04:17:52.086 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:51 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-26T04:17:52.086 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:51 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-26T04:17:52.086 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:51 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-26T04:17:52.086 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:51 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-26T04:17:52.086 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:51 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-26T04:17:52.086 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:51 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-26T04:17:52.086 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:51 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-26T04:17:52.087 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:51 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-26T04:17:52.087 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:51 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-26T04:17:52.087 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:51 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-26T04:17:52.087 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:51 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-26T04:17:52.087 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:51 smithi183 ceph-mon[33745]: Manager daemon y is now available 2024-03-26T04:17:52.087 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:51 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:17:52.087 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:51 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:17:52.087 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:51 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:17:52.087 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:51 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:17:52.087 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:51 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:17:52.087 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:51 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-26T04:17:52.087 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:51 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:17:52.087 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:51 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-26T04:17:53.137 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:52 smithi016 ceph-mon[37215]: mgrmap e20: y(active, since 1.01312s), standbys: x 2024-03-26T04:17:53.137 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:52 smithi016 ceph-mon[37215]: [26/Mar/2024:04:17:52] ENGINE Bus STARTING 2024-03-26T04:17:53.137 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:52 smithi016 ceph-mon[37215]: [26/Mar/2024:04:17:52] ENGINE Serving on http://172.21.15.16:8765 2024-03-26T04:17:53.138 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:52 smithi016 ceph-mon[37215]: [26/Mar/2024:04:17:52] ENGINE Serving on https://172.21.15.16:7150 2024-03-26T04:17:53.138 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:52 smithi016 ceph-mon[37215]: [26/Mar/2024:04:17:52] ENGINE Bus STARTED 2024-03-26T04:17:53.138 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:52 smithi016 ceph-mon[29101]: mgrmap e20: y(active, since 1.01312s), standbys: x 2024-03-26T04:17:53.138 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:52 smithi016 ceph-mon[29101]: [26/Mar/2024:04:17:52] ENGINE Bus STARTING 2024-03-26T04:17:53.138 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:52 smithi016 ceph-mon[29101]: [26/Mar/2024:04:17:52] ENGINE Serving on http://172.21.15.16:8765 2024-03-26T04:17:53.138 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:52 smithi016 ceph-mon[29101]: [26/Mar/2024:04:17:52] ENGINE Serving on https://172.21.15.16:7150 2024-03-26T04:17:53.138 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:52 smithi016 ceph-mon[29101]: [26/Mar/2024:04:17:52] ENGINE Bus STARTED 2024-03-26T04:17:53.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:52 smithi183 ceph-mon[33745]: mgrmap e20: y(active, since 1.01312s), standbys: x 2024-03-26T04:17:53.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:52 smithi183 ceph-mon[33745]: [26/Mar/2024:04:17:52] ENGINE Bus STARTING 2024-03-26T04:17:53.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:52 smithi183 ceph-mon[33745]: [26/Mar/2024:04:17:52] ENGINE Serving on http://172.21.15.16:8765 2024-03-26T04:17:53.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:52 smithi183 ceph-mon[33745]: [26/Mar/2024:04:17:52] ENGINE Serving on https://172.21.15.16:7150 2024-03-26T04:17:53.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:52 smithi183 ceph-mon[33745]: [26/Mar/2024:04:17:52] ENGINE Bus STARTED 2024-03-26T04:17:54.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:53 smithi183 ceph-mon[33745]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:17:54.189 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:53 smithi016 ceph-mon[37215]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:17:54.190 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:53 smithi016 ceph-mon[29101]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:17:55.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:54 smithi016 ceph-mon[37215]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-26T04:17:55.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:54 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:17:55.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:54 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:17:55.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:54 smithi016 ceph-mon[29101]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-26T04:17:55.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:54 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:17:55.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:54 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:17:55.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:54 smithi183 ceph-mon[33745]: mgrmap e21: y(active, since 3s), standbys: x 2024-03-26T04:17:55.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:54 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:17:55.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:54 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:17:56.416 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:17:56.503 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:56 smithi016 ceph-mon[37215]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:17:56.503 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:56 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:17:56.503 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:56 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:17:56.503 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:56 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:17:56.503 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:56 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:17:56.503 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:56 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:17:56.503 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:56 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:17:56.503 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:56 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:17:56.503 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:56 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:17:56.504 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:56 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:17:56.504 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:56 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:17:56.504 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:56 smithi016 ceph-mon[37215]: Adjusting osd_memory_target on smithi183 to 3760M 2024-03-26T04:17:56.504 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:56 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:17:56.504 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:56 smithi016 ceph-mon[37215]: mgrmap e22: y(active, since 5s), standbys: x 2024-03-26T04:17:56.504 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:56 smithi016 ceph-mon[29101]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:17:56.504 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:56 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:17:56.504 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:56 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:17:56.504 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:56 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:17:56.504 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:56 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:17:56.504 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:56 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:17:56.504 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:56 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:17:56.505 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:56 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:17:56.505 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:56 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:17:56.505 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:56 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:17:56.505 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:56 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:17:56.505 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:56 smithi016 ceph-mon[29101]: Adjusting osd_memory_target on smithi183 to 3760M 2024-03-26T04:17:56.505 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:56 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:17:56.505 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:56 smithi016 ceph-mon[29101]: mgrmap e22: y(active, since 5s), standbys: x 2024-03-26T04:17:56.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:56 smithi183 ceph-mon[33745]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:17:56.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:56 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:17:56.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:56 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:17:56.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:56 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:17:56.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:56 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:17:56.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:56 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:17:56.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:56 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:17:56.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:56 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:17:56.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:56 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:17:56.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:56 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:17:56.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:56 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:17:56.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:56 smithi183 ceph-mon[33745]: Adjusting osd_memory_target on smithi183 to 3760M 2024-03-26T04:17:56.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:56 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:17:56.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:56 smithi183 ceph-mon[33745]: mgrmap e22: y(active, since 5s), standbys: x 2024-03-26T04:17:57.447 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.445+0000 7f80cfbec700 1 -- 172.21.15.16:0/3742581499 >> v1:172.21.15.16:6790/0 conn(0x7f80c810e950 legacy=0x7f80c8110da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:57.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.446+0000 7f80cfbec700 1 -- 172.21.15.16:0/3742581499 shutdown_connections 2024-03-26T04:17:57.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.446+0000 7f80cfbec700 1 -- 172.21.15.16:0/3742581499 >> 172.21.15.16:0/3742581499 conn(0x7f80c80fc820 msgr2=0x7f80c80fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:17:57.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.446+0000 7f80cfbec700 1 -- 172.21.15.16:0/3742581499 shutdown_connections 2024-03-26T04:17:57.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.446+0000 7f80cfbec700 1 -- 172.21.15.16:0/3742581499 wait complete. 2024-03-26T04:17:57.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.446+0000 7f80cfbec700 1 Processor -- start 2024-03-26T04:17:57.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.446+0000 7f80cfbec700 1 -- start start 2024-03-26T04:17:57.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.446+0000 7f80cfbec700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f80c810a9a0 con 0x7f80c8103030 2024-03-26T04:17:57.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.446+0000 7f80cfbec700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f80c810ab70 con 0x7f80c810e950 2024-03-26T04:17:57.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.446+0000 7f80cfbec700 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f80c81a93a0 con 0x7f80c810ae40 2024-03-26T04:17:57.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.447+0000 7f80cd187700 1 --1- >> v1:172.21.15.16:6790/0 conn(0x7f80c810ae40 0x7f80c8107f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6790/0 says I am v1:172.21.15.16:38136/0 (socket says 172.21.15.16:38136) 2024-03-26T04:17:57.448 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.447+0000 7f80cd187700 1 -- 172.21.15.16:0/2473225890 learned_addr learned my addr 172.21.15.16:0/2473225890 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:17:57.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.447+0000 7f80beffd700 1 -- 172.21.15.16:0/2473225890 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 523610628 0 0) 0x7f80c81a93a0 con 0x7f80c810ae40 2024-03-26T04:17:57.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.447+0000 7f80beffd700 1 -- 172.21.15.16:0/2473225890 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80b40035b0 con 0x7f80c810ae40 2024-03-26T04:17:57.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.447+0000 7f80beffd700 1 -- 172.21.15.16:0/2473225890 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1947142336 0 0) 0x7f80c810ab70 con 0x7f80c810e950 2024-03-26T04:17:57.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.447+0000 7f80beffd700 1 -- 172.21.15.16:0/2473225890 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80c81a93a0 con 0x7f80c810e950 2024-03-26T04:17:57.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.447+0000 7f80beffd700 1 -- 172.21.15.16:0/2473225890 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2902738198 0 0) 0x7f80c810a9a0 con 0x7f80c8103030 2024-03-26T04:17:57.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.447+0000 7f80beffd700 1 -- 172.21.15.16:0/2473225890 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80c810ab70 con 0x7f80c8103030 2024-03-26T04:17:57.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.447+0000 7f80beffd700 1 -- 172.21.15.16:0/2473225890 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1654288438 0 0) 0x7f80b40035b0 con 0x7f80c810ae40 2024-03-26T04:17:57.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.447+0000 7f80beffd700 1 -- 172.21.15.16:0/2473225890 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80c810a9a0 con 0x7f80c810ae40 2024-03-26T04:17:57.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.447+0000 7f80beffd700 1 -- 172.21.15.16:0/2473225890 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1392525538 0 0) 0x7f80c81a93a0 con 0x7f80c810e950 2024-03-26T04:17:57.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.447+0000 7f80beffd700 1 -- 172.21.15.16:0/2473225890 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80b40035b0 con 0x7f80c810e950 2024-03-26T04:17:57.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.447+0000 7f80beffd700 1 -- 172.21.15.16:0/2473225890 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f80b80018b0 con 0x7f80c810ae40 2024-03-26T04:17:57.451 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.447+0000 7f80beffd700 1 -- 172.21.15.16:0/2473225890 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f80c0001c00 con 0x7f80c810e950 2024-03-26T04:17:57.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.447+0000 7f80beffd700 1 -- 172.21.15.16:0/2473225890 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2044851220 0 0) 0x7f80c810ab70 con 0x7f80c8103030 2024-03-26T04:17:57.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.447+0000 7f80beffd700 1 -- 172.21.15.16:0/2473225890 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80c81a93a0 con 0x7f80c8103030 2024-03-26T04:17:57.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.447+0000 7f80beffd700 1 -- 172.21.15.16:0/2473225890 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f80c40017f0 con 0x7f80c8103030 2024-03-26T04:17:57.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.447+0000 7f80beffd700 1 -- 172.21.15.16:0/2473225890 <== mon.2 v1:172.21.15.16:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3712892261 0 0) 0x7f80c810a9a0 con 0x7f80c810ae40 2024-03-26T04:17:57.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.447+0000 7f80beffd700 1 -- 172.21.15.16:0/2473225890 >> v1:172.21.15.183:6789/0 conn(0x7f80c810e950 legacy=0x7f80c81a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:57.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.447+0000 7f80beffd700 1 -- 172.21.15.16:0/2473225890 >> v1:172.21.15.16:6789/0 conn(0x7f80c8103030 legacy=0x7f80c81077f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:57.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.447+0000 7f80beffd700 1 -- 172.21.15.16:0/2473225890 --> v1:172.21.15.16:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80c81aa540 con 0x7f80c810ae40 2024-03-26T04:17:57.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.447+0000 7f80cfbec700 1 -- 172.21.15.16:0/2473225890 --> v1:172.21.15.16:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80c81a9590 con 0x7f80c810ae40 2024-03-26T04:17:57.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.447+0000 7f80cfbec700 1 -- 172.21.15.16:0/2473225890 --> v1:172.21.15.16:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f80c81a9a10 con 0x7f80c810ae40 2024-03-26T04:17:57.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.448+0000 7f80beffd700 1 -- 172.21.15.16:0/2473225890 <== mon.2 v1:172.21.15.16:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f80b8001ad0 con 0x7f80c810ae40 2024-03-26T04:17:57.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.448+0000 7f80beffd700 1 -- 172.21.15.16:0/2473225890 <== mon.2 v1:172.21.15.16:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f80b8005b80 con 0x7f80c810ae40 2024-03-26T04:17:57.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.448+0000 7f80cfbec700 1 -- 172.21.15.16:0/2473225890 --> v1:172.21.15.16:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f80ac000f80 con 0x7f80c810ae40 2024-03-26T04:17:57.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.449+0000 7f80beffd700 1 -- 172.21.15.16:0/2473225890 <== mon.2 v1:172.21.15.16:6790/0 7 ==== mgrmap(e 22) v1 ==== 91420+0+0 (unknown 3095259778 0 0) 0x7f80b8003950 con 0x7f80c810ae40 2024-03-26T04:17:57.452 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.449+0000 7f80beffd700 1 -- 172.21.15.16:0/2473225890 <== mon.2 v1:172.21.15.16:6790/0 8 ==== osd_map(70..70 src has 1..70) v4 ==== 6172+0+0 (unknown 1260882646 0 0) 0x7f80b8088860 con 0x7f80c810ae40 2024-03-26T04:17:57.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.450+0000 7f80beffd700 1 -- 172.21.15.16:0/2473225890 --> v1:172.21.15.16:6809/1107259766 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f80b407ce80 con 0x7f80b40791d0 2024-03-26T04:17:57.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.450+0000 7f80beffd700 1 -- 172.21.15.16:0/2473225890 <== mon.2 v1:172.21.15.16:6790/0 9 ==== mon_get_version_reply(handle=1 version=70) v2 ==== 24+0+0 (unknown 2350636211 0 0) 0x7f80b8088c90 con 0x7f80c810ae40 2024-03-26T04:17:57.453 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.450+0000 7f80beffd700 1 -- 172.21.15.16:0/2473225890 <== osd.2 v1:172.21.15.16:6809/1107259766 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f80b407ce80 con 0x7f80b40791d0 2024-03-26T04:17:57.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.467+0000 7f80cfbec700 1 -- 172.21.15.16:0/2473225890 --> v1:172.21.15.16:6809/1107259766 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f80ac002c40 con 0x7f80b40791d0 2024-03-26T04:17:57.468 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.467+0000 7f80beffd700 1 -- 172.21.15.16:0/2473225890 <== osd.2 v1:172.21.15.16:6809/1107259766 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 1652615586) 0x7f80ac002c40 con 0x7f80b40791d0 2024-03-26T04:17:57.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.467+0000 7f80cfbec700 1 -- 172.21.15.16:0/2473225890 >> v1:172.21.15.16:6809/1107259766 conn(0x7f80b40791d0 legacy=0x7f80b407b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:57.469 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.468+0000 7f80cfbec700 1 -- 172.21.15.16:0/2473225890 >> v1:172.21.15.16:6800/3868004154 conn(0x7f80b406ea50 legacy=0x7f80b4070f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:57.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.468+0000 7f80cfbec700 1 -- 172.21.15.16:0/2473225890 >> v1:172.21.15.16:6790/0 conn(0x7f80c810ae40 legacy=0x7f80c8107f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:57.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.468+0000 7f80ce189700 1 -- 172.21.15.16:0/2473225890 reap_dead start 2024-03-26T04:17:57.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.468+0000 7f80cfbec700 1 -- 172.21.15.16:0/2473225890 shutdown_connections 2024-03-26T04:17:57.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.468+0000 7f80cfbec700 1 -- 172.21.15.16:0/2473225890 >> 172.21.15.16:0/2473225890 conn(0x7f80c80fc820 msgr2=0x7f80c80fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:17:57.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.468+0000 7f80cfbec700 1 -- 172.21.15.16:0/2473225890 shutdown_connections 2024-03-26T04:17:57.470 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:57.468+0000 7f80cfbec700 1 -- 172.21.15.16:0/2473225890 wait complete. 2024-03-26T04:17:58.445 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:17:58 smithi016 ceph-mon[29101]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-03-26T04:17:58.446 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:17:58 smithi016 ceph-mon[37215]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-03-26T04:17:58.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:17:58 smithi183 ceph-mon[33745]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-03-26T04:17:58.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.815+0000 7fe0a14d2700 1 -- 172.21.15.16:0/946571732 >> v1:172.21.15.16:6790/0 conn(0x7fe09c10e8c0 legacy=0x7fe09c110d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:58.817 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.815+0000 7fe0a14d2700 1 -- 172.21.15.16:0/946571732 shutdown_connections 2024-03-26T04:17:58.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.815+0000 7fe0a14d2700 1 -- 172.21.15.16:0/946571732 >> 172.21.15.16:0/946571732 conn(0x7fe09c0fd960 msgr2=0x7fe09c0ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:17:58.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.815+0000 7fe0a14d2700 1 -- 172.21.15.16:0/946571732 shutdown_connections 2024-03-26T04:17:58.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.816+0000 7fe0a14d2700 1 -- 172.21.15.16:0/946571732 wait complete. 2024-03-26T04:17:58.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.816+0000 7fe0a14d2700 1 Processor -- start 2024-03-26T04:17:58.818 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.816+0000 7fe0a14d2700 1 -- start start 2024-03-26T04:17:58.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.816+0000 7fe0a14d2700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe09c1a6d20 con 0x7fe09c10e8c0 2024-03-26T04:17:58.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.816+0000 7fe0a14d2700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe09c1a7f00 con 0x7fe09c102090 2024-03-26T04:17:58.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.817+0000 7fe0a14d2700 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fe09c1a9100 con 0x7fe09c10ad20 2024-03-26T04:17:58.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.817+0000 7fe09a7fc700 1 --1- >> v1:172.21.15.16:6790/0 conn(0x7fe09c10ad20 0x7fe09c1a1c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6790/0 says I am v1:172.21.15.16:38152/0 (socket says 172.21.15.16:38152) 2024-03-26T04:17:58.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.817+0000 7fe09a7fc700 1 -- 172.21.15.16:0/3780739535 learned_addr learned my addr 172.21.15.16:0/3780739535 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:17:58.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.817+0000 7fe083fff700 1 -- 172.21.15.16:0/3780739535 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2006574117 0 0) 0x7fe09c1a6d20 con 0x7fe09c10e8c0 2024-03-26T04:17:58.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.817+0000 7fe083fff700 1 -- 172.21.15.16:0/3780739535 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0840049e0 con 0x7fe09c10e8c0 2024-03-26T04:17:58.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.817+0000 7fe083fff700 1 -- 172.21.15.16:0/3780739535 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3799278574 0 0) 0x7fe09c1a9100 con 0x7fe09c10ad20 2024-03-26T04:17:58.820 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.817+0000 7fe083fff700 1 -- 172.21.15.16:0/3780739535 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe09c1a6d20 con 0x7fe09c10ad20 2024-03-26T04:17:58.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.817+0000 7fe083fff700 1 -- 172.21.15.16:0/3780739535 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2984272867 0 0) 0x7fe09c1a7f00 con 0x7fe09c102090 2024-03-26T04:17:58.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.817+0000 7fe083fff700 1 -- 172.21.15.16:0/3780739535 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe09c1a9100 con 0x7fe09c102090 2024-03-26T04:17:58.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.817+0000 7fe083fff700 1 -- 172.21.15.16:0/3780739535 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3748379590 0 0) 0x7fe0840049e0 con 0x7fe09c10e8c0 2024-03-26T04:17:58.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.817+0000 7fe083fff700 1 -- 172.21.15.16:0/3780739535 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe09c1a7f00 con 0x7fe09c10e8c0 2024-03-26T04:17:58.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.817+0000 7fe083fff700 1 -- 172.21.15.16:0/3780739535 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fe094001cc0 con 0x7fe09c10e8c0 2024-03-26T04:17:58.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.817+0000 7fe083fff700 1 -- 172.21.15.16:0/3780739535 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 755487680 0 0) 0x7fe09c1a6d20 con 0x7fe09c10ad20 2024-03-26T04:17:58.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.817+0000 7fe083fff700 1 -- 172.21.15.16:0/3780739535 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0840049e0 con 0x7fe09c10ad20 2024-03-26T04:17:58.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.817+0000 7fe083fff700 1 -- 172.21.15.16:0/3780739535 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fe090001920 con 0x7fe09c10ad20 2024-03-26T04:17:58.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.817+0000 7fe083fff700 1 -- 172.21.15.16:0/3780739535 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4200038855 0 0) 0x7fe09c1a9100 con 0x7fe09c102090 2024-03-26T04:17:58.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.817+0000 7fe083fff700 1 -- 172.21.15.16:0/3780739535 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe09c1a6d20 con 0x7fe09c102090 2024-03-26T04:17:58.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.817+0000 7fe083fff700 1 -- 172.21.15.16:0/3780739535 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fe08c001870 con 0x7fe09c102090 2024-03-26T04:17:58.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.817+0000 7fe083fff700 1 -- 172.21.15.16:0/3780739535 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1521987387 0 0) 0x7fe09c1a7f00 con 0x7fe09c10e8c0 2024-03-26T04:17:58.821 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.817+0000 7fe083fff700 1 -- 172.21.15.16:0/3780739535 >> v1:172.21.15.16:6790/0 conn(0x7fe09c10ad20 legacy=0x7fe09c1a1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:58.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.817+0000 7fe083fff700 1 -- 172.21.15.16:0/3780739535 >> v1:172.21.15.183:6789/0 conn(0x7fe09c102090 legacy=0x7fe09c1072c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:58.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.817+0000 7fe083fff700 1 -- 172.21.15.16:0/3780739535 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe09c1aa300 con 0x7fe09c10e8c0 2024-03-26T04:17:58.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.817+0000 7fe0a14d2700 1 -- 172.21.15.16:0/3780739535 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe09c1a92f0 con 0x7fe09c10e8c0 2024-03-26T04:17:58.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.818+0000 7fe0a14d2700 1 -- 172.21.15.16:0/3780739535 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe09c1a9770 con 0x7fe09c10e8c0 2024-03-26T04:17:58.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.818+0000 7fe083fff700 1 -- 172.21.15.16:0/3780739535 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7fe0940049a0 con 0x7fe09c10e8c0 2024-03-26T04:17:58.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.818+0000 7fe083fff700 1 -- 172.21.15.16:0/3780739535 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fe094005050 con 0x7fe09c10e8c0 2024-03-26T04:17:58.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.818+0000 7fe0a14d2700 1 -- 172.21.15.16:0/3780739535 --> v1:172.21.15.16:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe088000f80 con 0x7fe09c10e8c0 2024-03-26T04:17:58.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.819+0000 7fe083fff700 1 -- 172.21.15.16:0/3780739535 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 22) v1 ==== 91420+0+0 (unknown 3095259778 0 0) 0x7fe094003a00 con 0x7fe09c10e8c0 2024-03-26T04:17:58.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.820+0000 7fe083fff700 1 -- 172.21.15.16:0/3780739535 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(70..70 src has 1..70) v4 ==== 6172+0+0 (unknown 1260882646 0 0) 0x7fe094087d20 con 0x7fe09c10e8c0 2024-03-26T04:17:58.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.820+0000 7fe083fff700 1 -- 172.21.15.16:0/3780739535 --> v1:172.21.15.16:6801/1132171550 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe08407cdf0 con 0x7fe084079140 2024-03-26T04:17:58.823 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.820+0000 7fe083fff700 1 -- 172.21.15.16:0/3780739535 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_get_version_reply(handle=1 version=70) v2 ==== 24+0+0 (unknown 2350636211 0 0) 0x7fe094088040 con 0x7fe09c10e8c0 2024-03-26T04:17:58.824 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.821+0000 7fe083fff700 1 -- 172.21.15.16:0/3780739535 <== osd.0 v1:172.21.15.16:6801/1132171550 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe08407cdf0 con 0x7fe084079140 2024-03-26T04:17:58.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.837+0000 7fe0a14d2700 1 -- 172.21.15.16:0/3780739535 --> v1:172.21.15.16:6801/1132171550 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe088002c40 con 0x7fe084079140 2024-03-26T04:17:58.840 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.838+0000 7fe083fff700 1 -- 172.21.15.16:0/3780739535 <== osd.0 v1:172.21.15.16:6801/1132171550 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 2546019811) 0x7fe088002c40 con 0x7fe084079140 2024-03-26T04:17:58.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.838+0000 7fe0a14d2700 1 -- 172.21.15.16:0/3780739535 >> v1:172.21.15.16:6801/1132171550 conn(0x7fe084079140 legacy=0x7fe08407b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:58.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.838+0000 7fe0a14d2700 1 -- 172.21.15.16:0/3780739535 >> v1:172.21.15.16:6800/3868004154 conn(0x7fe08406eac0 legacy=0x7fe084070f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:58.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.838+0000 7fe0a14d2700 1 -- 172.21.15.16:0/3780739535 >> v1:172.21.15.16:6789/0 conn(0x7fe09c10e8c0 legacy=0x7fe09c1a53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:17:58.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.838+0000 7fe09b7fe700 1 -- 172.21.15.16:0/3780739535 reap_dead start 2024-03-26T04:17:58.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.838+0000 7fe0a14d2700 1 -- 172.21.15.16:0/3780739535 shutdown_connections 2024-03-26T04:17:58.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.838+0000 7fe0a14d2700 1 -- 172.21.15.16:0/3780739535 >> 172.21.15.16:0/3780739535 conn(0x7fe09c0fd960 msgr2=0x7fe09c111a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:17:58.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.838+0000 7fe0a14d2700 1 -- 172.21.15.16:0/3780739535 shutdown_connections 2024-03-26T04:17:58.841 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:58.838+0000 7fe0a14d2700 1 -- 172.21.15.16:0/3780739535 wait complete. 2024-03-26T04:17:59.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:17:58 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:17:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:17:59.604 INFO:teuthology.orchestra.run.smithi016.stdout:34359738416 2024-03-26T04:17:59.605 DEBUG:teuthology.orchestra.run.smithi016:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-26T04:18:00.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.805+0000 7f96f3dac700 1 -- 172.21.15.16:0/1780730489 >> v1:172.21.15.16:6790/0 conn(0x7f96ec10e8a0 legacy=0x7f96ec110d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:00.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.805+0000 7f96f3dac700 1 -- 172.21.15.16:0/1780730489 shutdown_connections 2024-03-26T04:18:00.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.805+0000 7f96f3dac700 1 -- 172.21.15.16:0/1780730489 >> 172.21.15.16:0/1780730489 conn(0x7f96ec0771b0 msgr2=0x7f96ec0fe530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:18:00.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.805+0000 7f96f3dac700 1 -- 172.21.15.16:0/1780730489 shutdown_connections 2024-03-26T04:18:00.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.805+0000 7f96f3dac700 1 -- 172.21.15.16:0/1780730489 wait complete. 2024-03-26T04:18:00.064 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.805+0000 7f96f3dac700 1 Processor -- start 2024-03-26T04:18:00.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.805+0000 7f96f3dac700 1 -- start start 2024-03-26T04:18:00.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.806+0000 7f96f3dac700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f96ec1a6d10 con 0x7f96ec10e8a0 2024-03-26T04:18:00.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.806+0000 7f96f3dac700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f96ec1a7ef0 con 0x7f96ec100580 2024-03-26T04:18:00.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.806+0000 7f96f3dac700 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f96ec1a90f0 con 0x7f96ec10ad00 2024-03-26T04:18:00.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.806+0000 7f96f1347700 1 --1- >> v1:172.21.15.16:6790/0 conn(0x7f96ec10ad00 0x7f96ec1a1c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6790/0 says I am v1:172.21.15.16:38178/0 (socket says 172.21.15.16:38178) 2024-03-26T04:18:00.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.806+0000 7f96f1347700 1 -- 172.21.15.16:0/3423361277 learned_addr learned my addr 172.21.15.16:0/3423361277 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:18:00.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.806+0000 7f96e2ffd700 1 -- 172.21.15.16:0/3423361277 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 859031340 0 0) 0x7f96ec1a6d10 con 0x7f96ec10e8a0 2024-03-26T04:18:00.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.806+0000 7f96e2ffd700 1 -- 172.21.15.16:0/3423361277 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96d80035b0 con 0x7f96ec10e8a0 2024-03-26T04:18:00.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.806+0000 7f96e2ffd700 1 -- 172.21.15.16:0/3423361277 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2512087137 0 0) 0x7f96ec1a90f0 con 0x7f96ec10ad00 2024-03-26T04:18:00.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.806+0000 7f96e2ffd700 1 -- 172.21.15.16:0/3423361277 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96ec1a6d10 con 0x7f96ec10ad00 2024-03-26T04:18:00.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.806+0000 7f96e2ffd700 1 -- 172.21.15.16:0/3423361277 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2213601505 0 0) 0x7f96ec1a7ef0 con 0x7f96ec100580 2024-03-26T04:18:00.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.806+0000 7f96e2ffd700 1 -- 172.21.15.16:0/3423361277 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96ec1a90f0 con 0x7f96ec100580 2024-03-26T04:18:00.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.806+0000 7f96e2ffd700 1 -- 172.21.15.16:0/3423361277 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1673214180 0 0) 0x7f96ec1a6d10 con 0x7f96ec10ad00 2024-03-26T04:18:00.065 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.806+0000 7f96e2ffd700 1 -- 172.21.15.16:0/3423361277 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96ec1a7ef0 con 0x7f96ec10ad00 2024-03-26T04:18:00.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.806+0000 7f96e2ffd700 1 -- 172.21.15.16:0/3423361277 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2239721637 0 0) 0x7f96d80035b0 con 0x7f96ec10e8a0 2024-03-26T04:18:00.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.806+0000 7f96e2ffd700 1 -- 172.21.15.16:0/3423361277 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96ec1a6d10 con 0x7f96ec10e8a0 2024-03-26T04:18:00.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.806+0000 7f96e2ffd700 1 -- 172.21.15.16:0/3423361277 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 407828112 0 0) 0x7f96ec1a90f0 con 0x7f96ec100580 2024-03-26T04:18:00.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.806+0000 7f96e2ffd700 1 -- 172.21.15.16:0/3423361277 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96d80035b0 con 0x7f96ec100580 2024-03-26T04:18:00.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.806+0000 7f96e2ffd700 1 -- 172.21.15.16:0/3423361277 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f96e80018d0 con 0x7f96ec10ad00 2024-03-26T04:18:00.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.806+0000 7f96e2ffd700 1 -- 172.21.15.16:0/3423361277 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f96e40017b0 con 0x7f96ec10e8a0 2024-03-26T04:18:00.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.806+0000 7f96e2ffd700 1 -- 172.21.15.16:0/3423361277 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f96dc001830 con 0x7f96ec100580 2024-03-26T04:18:00.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.807+0000 7f96e2ffd700 1 -- 172.21.15.16:0/3423361277 <== mon.2 v1:172.21.15.16:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2710423431 0 0) 0x7f96ec1a7ef0 con 0x7f96ec10ad00 2024-03-26T04:18:00.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.807+0000 7f96e2ffd700 1 -- 172.21.15.16:0/3423361277 >> v1:172.21.15.183:6789/0 conn(0x7f96ec100580 legacy=0x7f96ec1072b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:00.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.807+0000 7f96e2ffd700 1 -- 172.21.15.16:0/3423361277 >> v1:172.21.15.16:6789/0 conn(0x7f96ec10e8a0 legacy=0x7f96ec1a53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:00.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.807+0000 7f96e2ffd700 1 -- 172.21.15.16:0/3423361277 --> v1:172.21.15.16:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96ec1aa2f0 con 0x7f96ec10ad00 2024-03-26T04:18:00.066 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.807+0000 7f96f3dac700 1 -- 172.21.15.16:0/3423361277 --> v1:172.21.15.16:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96ec1a6f00 con 0x7f96ec10ad00 2024-03-26T04:18:00.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.807+0000 7f96f3dac700 1 -- 172.21.15.16:0/3423361277 --> v1:172.21.15.16:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f96ec1a7380 con 0x7f96ec10ad00 2024-03-26T04:18:00.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.807+0000 7f96e2ffd700 1 -- 172.21.15.16:0/3423361277 <== mon.2 v1:172.21.15.16:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f96e8003b10 con 0x7f96ec10ad00 2024-03-26T04:18:00.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.807+0000 7f96e2ffd700 1 -- 172.21.15.16:0/3423361277 <== mon.2 v1:172.21.15.16:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f96e8005c30 con 0x7f96ec10ad00 2024-03-26T04:18:00.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.807+0000 7f96f3dac700 1 -- 172.21.15.16:0/3423361277 --> v1:172.21.15.16:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f96d0000f80 con 0x7f96ec10ad00 2024-03-26T04:18:00.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.808+0000 7f96e2ffd700 1 -- 172.21.15.16:0/3423361277 <== mon.2 v1:172.21.15.16:6790/0 7 ==== mgrmap(e 22) v1 ==== 91420+0+0 (unknown 3095259778 0 0) 0x7f96e8003680 con 0x7f96ec10ad00 2024-03-26T04:18:00.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.809+0000 7f96e2ffd700 1 -- 172.21.15.16:0/3423361277 <== mon.2 v1:172.21.15.16:6790/0 8 ==== osd_map(70..70 src has 1..70) v4 ==== 6172+0+0 (unknown 1260882646 0 0) 0x7f96e8088700 con 0x7f96ec10ad00 2024-03-26T04:18:00.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.809+0000 7f96e2ffd700 1 -- 172.21.15.16:0/3423361277 --> v1:172.21.15.183:6812/804876304 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f96d807d000 con 0x7f96d8079350 2024-03-26T04:18:00.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.809+0000 7f96e2ffd700 1 -- 172.21.15.16:0/3423361277 <== mon.2 v1:172.21.15.16:6790/0 9 ==== mon_get_version_reply(handle=1 version=70) v2 ==== 24+0+0 (unknown 2350636211 0 0) 0x7f96e8088b00 con 0x7f96ec10ad00 2024-03-26T04:18:00.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.810+0000 7f96e2ffd700 1 -- 172.21.15.16:0/3423361277 <== osd.7 v1:172.21.15.183:6812/804876304 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f96d807d000 con 0x7f96d8079350 2024-03-26T04:18:00.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.828+0000 7f96f3dac700 1 -- 172.21.15.16:0/3423361277 --> v1:172.21.15.183:6812/804876304 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f96d0002c40 con 0x7f96d8079350 2024-03-26T04:18:00.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.828+0000 7f96e2ffd700 1 -- 172.21.15.16:0/3423361277 <== osd.7 v1:172.21.15.183:6812/804876304 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (unknown 0 0 182067233) 0x7f96d0002c40 con 0x7f96d8079350 2024-03-26T04:18:00.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.829+0000 7f96f3dac700 1 -- 172.21.15.16:0/3423361277 >> v1:172.21.15.183:6812/804876304 conn(0x7f96d8079350 legacy=0x7f96d807b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:00.067 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.829+0000 7f96f3dac700 1 -- 172.21.15.16:0/3423361277 >> v1:172.21.15.16:6800/3868004154 conn(0x7f96d806ec50 legacy=0x7f96d8071110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:00.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.829+0000 7f96f3dac700 1 -- 172.21.15.16:0/3423361277 >> v1:172.21.15.16:6790/0 conn(0x7f96ec10ad00 legacy=0x7f96ec1a1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:00.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.829+0000 7f96f2349700 1 -- 172.21.15.16:0/3423361277 reap_dead start 2024-03-26T04:18:00.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.829+0000 7f96f3dac700 1 -- 172.21.15.16:0/3423361277 shutdown_connections 2024-03-26T04:18:00.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.829+0000 7f96f3dac700 1 -- 172.21.15.16:0/3423361277 >> 172.21.15.16:0/3423361277 conn(0x7f96ec0771b0 msgr2=0x7f96ec111510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:18:00.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.829+0000 7f96f3dac700 1 -- 172.21.15.16:0/3423361277 shutdown_connections 2024-03-26T04:18:00.068 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:17:59.829+0000 7f96f3dac700 1 -- 172.21.15.16:0/3423361277 wait complete. 2024-03-26T04:18:00.357 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:00 smithi016 ceph-mon[37215]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-26T04:18:00.358 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:00 smithi016 ceph-mon[29101]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-26T04:18:00.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:00 smithi183 ceph-mon[33745]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-26T04:18:01.131 INFO:teuthology.orchestra.run.smithi016.stdout:77309411367 2024-03-26T04:18:01.131 DEBUG:teuthology.orchestra.run.smithi016:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-26T04:18:01.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.337+0000 7fc6dd1e9700 1 -- 172.21.15.16:0/918407658 >> v1:172.21.15.16:6790/0 conn(0x7fc6d810e990 legacy=0x7fc6d8110e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:01.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.337+0000 7fc6dd1e9700 1 -- 172.21.15.16:0/918407658 shutdown_connections 2024-03-26T04:18:01.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.337+0000 7fc6dd1e9700 1 -- 172.21.15.16:0/918407658 >> 172.21.15.16:0/918407658 conn(0x7fc6d80771b0 msgr2=0x7fc6d80fe530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:18:01.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.337+0000 7fc6dd1e9700 1 -- 172.21.15.16:0/918407658 shutdown_connections 2024-03-26T04:18:01.339 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.338+0000 7fc6dd1e9700 1 -- 172.21.15.16:0/918407658 wait complete. 2024-03-26T04:18:01.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.339+0000 7fc6dd1e9700 1 Processor -- start 2024-03-26T04:18:01.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.339+0000 7fc6dd1e9700 1 -- start start 2024-03-26T04:18:01.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.339+0000 7fc6dd1e9700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc6d80ff7c0 con 0x7fc6d810acc0 2024-03-26T04:18:01.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.339+0000 7fc6dd1e9700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc6d80ff990 con 0x7fc6d810e990 2024-03-26T04:18:01.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.339+0000 7fc6dd1e9700 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc6d81a9200 con 0x7fc6d8106e40 2024-03-26T04:18:01.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.340+0000 7fc6d6d9d700 1 --1- >> v1:172.21.15.16:6790/0 conn(0x7fc6d8106e40 0x7fc6d80fc610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6790/0 says I am v1:172.21.15.16:38198/0 (socket says 172.21.15.16:38198) 2024-03-26T04:18:01.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.340+0000 7fc6d6d9d700 1 -- 172.21.15.16:0/2137596756 learned_addr learned my addr 172.21.15.16:0/2137596756 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:18:01.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.340+0000 7fc6bffff700 1 -- 172.21.15.16:0/2137596756 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 806743370 0 0) 0x7fc6d81a9200 con 0x7fc6d8106e40 2024-03-26T04:18:01.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.340+0000 7fc6bffff700 1 -- 172.21.15.16:0/2137596756 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6c00031f0 con 0x7fc6d8106e40 2024-03-26T04:18:01.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.340+0000 7fc6bffff700 1 -- 172.21.15.16:0/2137596756 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1759744662 0 0) 0x7fc6d80ff990 con 0x7fc6d810e990 2024-03-26T04:18:01.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.340+0000 7fc6bffff700 1 -- 172.21.15.16:0/2137596756 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6d81a9200 con 0x7fc6d810e990 2024-03-26T04:18:01.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.340+0000 7fc6bffff700 1 -- 172.21.15.16:0/2137596756 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1689513604 0 0) 0x7fc6c00031f0 con 0x7fc6d8106e40 2024-03-26T04:18:01.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.340+0000 7fc6bffff700 1 -- 172.21.15.16:0/2137596756 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6d80ff990 con 0x7fc6d8106e40 2024-03-26T04:18:01.342 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.340+0000 7fc6bffff700 1 -- 172.21.15.16:0/2137596756 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fc6c80019f0 con 0x7fc6d8106e40 2024-03-26T04:18:01.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.340+0000 7fc6bffff700 1 -- 172.21.15.16:0/2137596756 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3476472213 0 0) 0x7fc6d81a9200 con 0x7fc6d810e990 2024-03-26T04:18:01.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.340+0000 7fc6bffff700 1 -- 172.21.15.16:0/2137596756 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6c00031f0 con 0x7fc6d810e990 2024-03-26T04:18:01.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.340+0000 7fc6bffff700 1 -- 172.21.15.16:0/2137596756 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fc6d00046e0 con 0x7fc6d810e990 2024-03-26T04:18:01.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.340+0000 7fc6bffff700 1 -- 172.21.15.16:0/2137596756 <== mon.2 v1:172.21.15.16:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2194401071 0 0) 0x7fc6d80ff990 con 0x7fc6d8106e40 2024-03-26T04:18:01.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.341+0000 7fc6bffff700 1 -- 172.21.15.16:0/2137596756 >> v1:172.21.15.183:6789/0 conn(0x7fc6d810e990 legacy=0x7fc6d81a5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:01.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.341+0000 7fc6bffff700 1 -- 172.21.15.16:0/2137596756 >> v1:172.21.15.16:6789/0 conn(0x7fc6d810acc0 legacy=0x7fc6d80fcd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:01.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.341+0000 7fc6bffff700 1 -- 172.21.15.16:0/2137596756 --> v1:172.21.15.16:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d81aa3a0 con 0x7fc6d8106e40 2024-03-26T04:18:01.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.341+0000 7fc6dd1e9700 1 -- 172.21.15.16:0/2137596756 --> v1:172.21.15.16:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6d81a93f0 con 0x7fc6d8106e40 2024-03-26T04:18:01.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.341+0000 7fc6bffff700 1 -- 172.21.15.16:0/2137596756 <== mon.2 v1:172.21.15.16:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7fc6c8003b10 con 0x7fc6d8106e40 2024-03-26T04:18:01.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.341+0000 7fc6bffff700 1 -- 172.21.15.16:0/2137596756 <== mon.2 v1:172.21.15.16:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fc6c8005b90 con 0x7fc6d8106e40 2024-03-26T04:18:01.343 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.341+0000 7fc6dd1e9700 1 -- 172.21.15.16:0/2137596756 --> v1:172.21.15.16:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6d81a9870 con 0x7fc6d8106e40 2024-03-26T04:18:01.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.342+0000 7fc6dd1e9700 1 -- 172.21.15.16:0/2137596756 --> v1:172.21.15.16:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc6d81149b0 con 0x7fc6d8106e40 2024-03-26T04:18:01.344 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.343+0000 7fc6bffff700 1 -- 172.21.15.16:0/2137596756 <== mon.2 v1:172.21.15.16:6790/0 7 ==== mgrmap(e 22) v1 ==== 91420+0+0 (unknown 3095259778 0 0) 0x7fc6c8005e20 con 0x7fc6d8106e40 2024-03-26T04:18:01.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.343+0000 7fc6bffff700 1 -- 172.21.15.16:0/2137596756 <== mon.2 v1:172.21.15.16:6790/0 8 ==== osd_map(70..70 src has 1..70) v4 ==== 6172+0+0 (unknown 1260882646 0 0) 0x7fc6c80889e0 con 0x7fc6d8106e40 2024-03-26T04:18:01.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.343+0000 7fc6bffff700 1 -- 172.21.15.16:0/2137596756 --> v1:172.21.15.16:6805/317987982 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc6c0081230 con 0x7fc6c007d580 2024-03-26T04:18:01.345 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.343+0000 7fc6bffff700 1 -- 172.21.15.16:0/2137596756 <== mon.2 v1:172.21.15.16:6790/0 9 ==== mon_get_version_reply(handle=1 version=70) v2 ==== 24+0+0 (unknown 2350636211 0 0) 0x7fc6c8088e10 con 0x7fc6d8106e40 2024-03-26T04:18:01.346 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.345+0000 7fc6bffff700 1 -- 172.21.15.16:0/2137596756 <== osd.1 v1:172.21.15.16:6805/317987982 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fc6d0008e90 con 0x7fc6c007d580 2024-03-26T04:18:01.364 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.363+0000 7fc6dd1e9700 1 -- 172.21.15.16:0/2137596756 --> v1:172.21.15.16:6805/317987982 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc6d81a9db0 con 0x7fc6c007d580 2024-03-26T04:18:01.365 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.364+0000 7fc6bffff700 1 -- 172.21.15.16:0/2137596756 <== osd.1 v1:172.21.15.16:6805/317987982 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 4200525082) 0x7fc6d81a9db0 con 0x7fc6c007d580 2024-03-26T04:18:01.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.364+0000 7fc6dd1e9700 1 -- 172.21.15.16:0/2137596756 >> v1:172.21.15.16:6805/317987982 conn(0x7fc6c007d580 legacy=0x7fc6c007f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:01.366 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.364+0000 7fc6dd1e9700 1 -- 172.21.15.16:0/2137596756 >> v1:172.21.15.16:6800/3868004154 conn(0x7fc6c0072df0 legacy=0x7fc6c00752b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:01.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.364+0000 7fc6dd1e9700 1 -- 172.21.15.16:0/2137596756 >> v1:172.21.15.16:6790/0 conn(0x7fc6d8106e40 legacy=0x7fc6d80fc610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:01.371 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.364+0000 7fc6d759e700 1 -- 172.21.15.16:0/2137596756 reap_dead start 2024-03-26T04:18:01.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.364+0000 7fc6dd1e9700 1 -- 172.21.15.16:0/2137596756 shutdown_connections 2024-03-26T04:18:01.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.364+0000 7fc6dd1e9700 1 -- 172.21.15.16:0/2137596756 >> 172.21.15.16:0/2137596756 conn(0x7fc6d80771b0 msgr2=0x7fc6d81120c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:18:01.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.364+0000 7fc6dd1e9700 1 -- 172.21.15.16:0/2137596756 shutdown_connections 2024-03-26T04:18:01.372 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:01.364+0000 7fc6dd1e9700 1 -- 172.21.15.16:0/2137596756 wait complete. 2024-03-26T04:18:02.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:18:01 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:18:02.387 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:02 smithi016 ceph-mon[37215]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-26T04:18:02.388 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:02 smithi016 ceph-mon[29101]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-26T04:18:02.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:02 smithi183 ceph-mon[33745]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 24 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-26T04:18:03.155 INFO:teuthology.orchestra.run.smithi016.stdout:231928233998 2024-03-26T04:18:03.155 DEBUG:teuthology.orchestra.run.smithi016:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-26T04:18:03.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:03 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:18:03.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:03 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:18:03.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:03.771+0000 7f8bb0952700 1 -- 172.21.15.16:0/2434105355 >> v1:172.21.15.16:6790/0 conn(0x7f8bac10d7a0 legacy=0x7f8bac10fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:03.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:03.771+0000 7f8bb0952700 1 -- 172.21.15.16:0/2434105355 shutdown_connections 2024-03-26T04:18:03.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:03.771+0000 7f8bb0952700 1 -- 172.21.15.16:0/2434105355 >> 172.21.15.16:0/2434105355 conn(0x7f8bac0fc820 msgr2=0x7f8bac0fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:18:03.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:03.771+0000 7f8bb0952700 1 -- 172.21.15.16:0/2434105355 shutdown_connections 2024-03-26T04:18:03.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:03.771+0000 7f8bb0952700 1 -- 172.21.15.16:0/2434105355 wait complete. 2024-03-26T04:18:03.772 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:03.772+0000 7f8bb0952700 1 Processor -- start 2024-03-26T04:18:03.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:03.772+0000 7f8bb0952700 1 -- start start 2024-03-26T04:18:03.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:03.772+0000 7f8bb0952700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8bac19dcd0 con 0x7f8bac100f70 2024-03-26T04:18:03.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:03.772+0000 7f8bb0952700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8bac1ae550 con 0x7f8bac109c00 2024-03-26T04:18:03.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:03.772+0000 7f8bb0952700 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8bac1af750 con 0x7f8bac10d7a0 2024-03-26T04:18:03.773 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:03.773+0000 7f8baa59c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8bac100f70 0x7f8bac106040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55542/0 (socket says 172.21.15.16:55542) 2024-03-26T04:18:03.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:03.773+0000 7f8baa59c700 1 -- 172.21.15.16:0/3719857182 learned_addr learned my addr 172.21.15.16:0/3719857182 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:18:03.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:03.773+0000 7f8ba37fe700 1 -- 172.21.15.16:0/3719857182 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3867105525 0 0) 0x7f8bac19dcd0 con 0x7f8bac100f70 2024-03-26T04:18:03.774 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:03.773+0000 7f8ba37fe700 1 -- 172.21.15.16:0/3719857182 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b980035b0 con 0x7f8bac100f70 2024-03-26T04:18:03.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:03.773+0000 7f8ba37fe700 1 -- 172.21.15.16:0/3719857182 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3610396571 0 0) 0x7f8b980035b0 con 0x7f8bac100f70 2024-03-26T04:18:03.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:03.773+0000 7f8ba37fe700 1 -- 172.21.15.16:0/3719857182 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8bac19dcd0 con 0x7f8bac100f70 2024-03-26T04:18:03.775 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:03.773+0000 7f8ba37fe700 1 -- 172.21.15.16:0/3719857182 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f8b9c001cf0 con 0x7f8bac100f70 2024-03-26T04:18:03.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:03.774+0000 7f8ba37fe700 1 -- 172.21.15.16:0/3719857182 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2573281421 0 0) 0x7f8bac19dcd0 con 0x7f8bac100f70 2024-03-26T04:18:03.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:03.774+0000 7f8ba37fe700 1 -- 172.21.15.16:0/3719857182 >> v1:172.21.15.16:6790/0 conn(0x7f8bac10d7a0 legacy=0x7f8bac1abe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:03.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:03.774+0000 7f8ba37fe700 1 -- 172.21.15.16:0/3719857182 >> v1:172.21.15.183:6789/0 conn(0x7f8bac109c00 legacy=0x7f8bac1a86f0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-26T04:18:03.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:03.774+0000 7f8ba37fe700 1 -- 172.21.15.16:0/3719857182 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bac1b0950 con 0x7f8bac100f70 2024-03-26T04:18:03.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:03.774+0000 7f8bb0952700 1 -- 172.21.15.16:0/3719857182 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bac1af940 con 0x7f8bac100f70 2024-03-26T04:18:03.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:03.774+0000 7f8bb0952700 1 -- 172.21.15.16:0/3719857182 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bac1afdc0 con 0x7f8bac100f70 2024-03-26T04:18:03.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:03.774+0000 7f8ba37fe700 1 -- 172.21.15.16:0/3719857182 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f8b9c002090 con 0x7f8bac100f70 2024-03-26T04:18:03.776 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:03.774+0000 7f8ba37fe700 1 -- 172.21.15.16:0/3719857182 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f8b9c0060c0 con 0x7f8bac100f70 2024-03-26T04:18:03.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:03.776+0000 7f8bb0952700 1 -- 172.21.15.16:0/3719857182 --> v1:172.21.15.16:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8b8c000f80 con 0x7f8bac100f70 2024-03-26T04:18:03.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:03.776+0000 7f8ba37fe700 1 -- 172.21.15.16:0/3719857182 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 22) v1 ==== 91420+0+0 (unknown 3095259778 0 0) 0x7f8b9c01c750 con 0x7f8bac100f70 2024-03-26T04:18:03.777 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:03.776+0000 7f8ba37fe700 1 -- 172.21.15.16:0/3719857182 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(70..70 src has 1..70) v4 ==== 6172+0+0 (unknown 1260882646 0 0) 0x7f8b9c088cf0 con 0x7f8bac100f70 2024-03-26T04:18:03.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:03.777+0000 7f8ba37fe700 1 -- 172.21.15.16:0/3719857182 --> v1:172.21.15.16:6813/1463789444 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8b9807ce10 con 0x7f8b98079160 2024-03-26T04:18:03.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:03.777+0000 7f8ba37fe700 1 -- 172.21.15.16:0/3719857182 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_get_version_reply(handle=1 version=70) v2 ==== 24+0+0 (unknown 2350636211 0 0) 0x7f8b9c089120 con 0x7f8bac100f70 2024-03-26T04:18:03.778 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:03.778+0000 7f8ba37fe700 1 -- 172.21.15.16:0/3719857182 <== osd.3 v1:172.21.15.16:6813/1463789444 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f8b9807ce10 con 0x7f8b98079160 2024-03-26T04:18:03.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:03 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:18:03.794 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:03.794+0000 7f8bb0952700 1 -- 172.21.15.16:0/3719857182 --> v1:172.21.15.16:6813/1463789444 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8b8c002c40 con 0x7f8b98079160 2024-03-26T04:18:03.795 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:03.795+0000 7f8ba37fe700 1 -- 172.21.15.16:0/3719857182 <== osd.3 v1:172.21.15.16:6813/1463789444 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (unknown 0 0 3775874274) 0x7f8b8c002c40 con 0x7f8b98079160 2024-03-26T04:18:03.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:03.795+0000 7f8bb0952700 1 -- 172.21.15.16:0/3719857182 >> v1:172.21.15.16:6813/1463789444 conn(0x7f8b98079160 legacy=0x7f8b9807b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:03.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:03.795+0000 7f8bb0952700 1 -- 172.21.15.16:0/3719857182 >> v1:172.21.15.16:6800/3868004154 conn(0x7f8b9806e760 legacy=0x7f8b98070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:03.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:03.795+0000 7f8bb0952700 1 -- 172.21.15.16:0/3719857182 >> v1:172.21.15.16:6789/0 conn(0x7f8bac100f70 legacy=0x7f8bac106040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:03.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:03.795+0000 7f8baad9d700 1 -- 172.21.15.16:0/3719857182 reap_dead start 2024-03-26T04:18:03.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:03.795+0000 7f8bb0952700 1 -- 172.21.15.16:0/3719857182 shutdown_connections 2024-03-26T04:18:03.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:03.795+0000 7f8bb0952700 1 -- 172.21.15.16:0/3719857182 >> 172.21.15.16:0/3719857182 conn(0x7f8bac0fc820 msgr2=0x7f8bac06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:18:03.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:03.795+0000 7f8bb0952700 1 -- 172.21.15.16:0/3719857182 shutdown_connections 2024-03-26T04:18:03.796 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:03.795+0000 7f8bb0952700 1 -- 172.21.15.16:0/3719857182 wait complete. 2024-03-26T04:18:04.521 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:04 smithi016 ceph-mon[29101]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-26T04:18:04.522 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:04 smithi016 ceph-mon[37215]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-26T04:18:04.548 INFO:teuthology.orchestra.run.smithi016.stdout:55834574892 2024-03-26T04:18:04.549 DEBUG:teuthology.orchestra.run.smithi016:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-26T04:18:04.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:04 smithi183 ceph-mon[33745]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-03-26T04:18:04.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.876+0000 7f5aa2d28700 1 -- 172.21.15.16:0/1438495706 >> v1:172.21.15.16:6790/0 conn(0x7f5a9c10ad10 legacy=0x7f5a9c10d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:04.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.876+0000 7f5aa2d28700 1 -- 172.21.15.16:0/1438495706 shutdown_connections 2024-03-26T04:18:04.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.876+0000 7f5aa2d28700 1 -- 172.21.15.16:0/1438495706 >> 172.21.15.16:0/1438495706 conn(0x7f5a9c0fd930 msgr2=0x7f5a9c0ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:18:04.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.876+0000 7f5aa2d28700 1 -- 172.21.15.16:0/1438495706 shutdown_connections 2024-03-26T04:18:04.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.876+0000 7f5aa2d28700 1 -- 172.21.15.16:0/1438495706 wait complete. 2024-03-26T04:18:04.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.876+0000 7f5aa2d28700 1 Processor -- start 2024-03-26T04:18:04.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.876+0000 7f5aa2d28700 1 -- start start 2024-03-26T04:18:04.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.877+0000 7f5aa2d28700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5a9c1a6cd0 con 0x7f5a9c10e8b0 2024-03-26T04:18:04.886 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.877+0000 7f5aa2d28700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5a9c1a7eb0 con 0x7f5a9c102080 2024-03-26T04:18:04.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.877+0000 7f5aa2d28700 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5a9c1a90b0 con 0x7f5a9c10ad10 2024-03-26T04:18:04.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.877+0000 7f5aa12c5700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5a9c10e8b0 0x7f5a9c1a5390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:55568/0 (socket says 172.21.15.16:55568) 2024-03-26T04:18:04.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.877+0000 7f5aa12c5700 1 -- 172.21.15.16:0/4073972219 learned_addr learned my addr 172.21.15.16:0/4073972219 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:18:04.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.877+0000 7f5a99ffb700 1 -- 172.21.15.16:0/4073972219 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 629800153 0 0) 0x7f5a9c1a6cd0 con 0x7f5a9c10e8b0 2024-03-26T04:18:04.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.877+0000 7f5a99ffb700 1 -- 172.21.15.16:0/4073972219 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a8c0035b0 con 0x7f5a9c10e8b0 2024-03-26T04:18:04.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.877+0000 7f5a99ffb700 1 -- 172.21.15.16:0/4073972219 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3058556738 0 0) 0x7f5a9c1a90b0 con 0x7f5a9c10ad10 2024-03-26T04:18:04.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.877+0000 7f5a99ffb700 1 -- 172.21.15.16:0/4073972219 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a9c1a6cd0 con 0x7f5a9c10ad10 2024-03-26T04:18:04.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.877+0000 7f5a99ffb700 1 -- 172.21.15.16:0/4073972219 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3218501928 0 0) 0x7f5a9c1a7eb0 con 0x7f5a9c102080 2024-03-26T04:18:04.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.877+0000 7f5a99ffb700 1 -- 172.21.15.16:0/4073972219 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a9c1a90b0 con 0x7f5a9c102080 2024-03-26T04:18:04.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.877+0000 7f5a99ffb700 1 -- 172.21.15.16:0/4073972219 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2598869815 0 0) 0x7f5a9c1a6cd0 con 0x7f5a9c10ad10 2024-03-26T04:18:04.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.877+0000 7f5a99ffb700 1 -- 172.21.15.16:0/4073972219 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a9c1a7eb0 con 0x7f5a9c10ad10 2024-03-26T04:18:04.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.877+0000 7f5a99ffb700 1 -- 172.21.15.16:0/4073972219 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1942458679 0 0) 0x7f5a8c0035b0 con 0x7f5a9c10e8b0 2024-03-26T04:18:04.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.877+0000 7f5a99ffb700 1 -- 172.21.15.16:0/4073972219 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a9c1a6cd0 con 0x7f5a9c10e8b0 2024-03-26T04:18:04.887 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.877+0000 7f5a99ffb700 1 -- 172.21.15.16:0/4073972219 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f5a88001360 con 0x7f5a9c10ad10 2024-03-26T04:18:04.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.877+0000 7f5a99ffb700 1 -- 172.21.15.16:0/4073972219 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f5a94001c50 con 0x7f5a9c10e8b0 2024-03-26T04:18:04.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.877+0000 7f5a99ffb700 1 -- 172.21.15.16:0/4073972219 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 874401504 0 0) 0x7f5a9c1a90b0 con 0x7f5a9c102080 2024-03-26T04:18:04.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.877+0000 7f5a99ffb700 1 -- 172.21.15.16:0/4073972219 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a8c0035b0 con 0x7f5a9c102080 2024-03-26T04:18:04.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.877+0000 7f5a99ffb700 1 -- 172.21.15.16:0/4073972219 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f5a90001830 con 0x7f5a9c102080 2024-03-26T04:18:04.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.878+0000 7f5a99ffb700 1 -- 172.21.15.16:0/4073972219 <== mon.2 v1:172.21.15.16:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2381255371 0 0) 0x7f5a9c1a7eb0 con 0x7f5a9c10ad10 2024-03-26T04:18:04.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.878+0000 7f5a99ffb700 1 -- 172.21.15.16:0/4073972219 >> v1:172.21.15.183:6789/0 conn(0x7f5a9c102080 legacy=0x7f5a9c107270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:04.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.878+0000 7f5a99ffb700 1 -- 172.21.15.16:0/4073972219 >> v1:172.21.15.16:6789/0 conn(0x7f5a9c10e8b0 legacy=0x7f5a9c1a5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:04.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.878+0000 7f5a99ffb700 1 -- 172.21.15.16:0/4073972219 --> v1:172.21.15.16:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a9c1aa2b0 con 0x7f5a9c10ad10 2024-03-26T04:18:04.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.878+0000 7f5aa2d28700 1 -- 172.21.15.16:0/4073972219 --> v1:172.21.15.16:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a9c1a92a0 con 0x7f5a9c10ad10 2024-03-26T04:18:04.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.878+0000 7f5aa2d28700 1 -- 172.21.15.16:0/4073972219 --> v1:172.21.15.16:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a9c1a9720 con 0x7f5a9c10ad10 2024-03-26T04:18:04.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.878+0000 7f5a99ffb700 1 -- 172.21.15.16:0/4073972219 <== mon.2 v1:172.21.15.16:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f5a880019a0 con 0x7f5a9c10ad10 2024-03-26T04:18:04.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.878+0000 7f5a99ffb700 1 -- 172.21.15.16:0/4073972219 <== mon.2 v1:172.21.15.16:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f5a88004d20 con 0x7f5a9c10ad10 2024-03-26T04:18:04.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.879+0000 7f5aa2d28700 1 -- 172.21.15.16:0/4073972219 --> v1:172.21.15.16:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5a80000f80 con 0x7f5a9c10ad10 2024-03-26T04:18:04.888 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.879+0000 7f5a99ffb700 1 -- 172.21.15.16:0/4073972219 <== mon.2 v1:172.21.15.16:6790/0 7 ==== mgrmap(e 22) v1 ==== 91420+0+0 (unknown 3095259778 0 0) 0x7f5a88003680 con 0x7f5a9c10ad10 2024-03-26T04:18:04.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.880+0000 7f5a99ffb700 1 -- 172.21.15.16:0/4073972219 <== mon.2 v1:172.21.15.16:6790/0 8 ==== osd_map(70..70 src has 1..70) v4 ==== 6172+0+0 (unknown 1260882646 0 0) 0x7f5a88087910 con 0x7f5a9c10ad10 2024-03-26T04:18:04.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.880+0000 7f5a99ffb700 1 -- 172.21.15.16:0/4073972219 --> v1:172.21.15.183:6804/3356204259 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5a8c07cd70 con 0x7f5a8c0790c0 2024-03-26T04:18:04.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.880+0000 7f5a99ffb700 1 -- 172.21.15.16:0/4073972219 <== mon.2 v1:172.21.15.16:6790/0 9 ==== mon_get_version_reply(handle=1 version=70) v2 ==== 24+0+0 (unknown 2350636211 0 0) 0x7f5a88087cb0 con 0x7f5a9c10ad10 2024-03-26T04:18:04.889 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.881+0000 7f5a99ffb700 1 -- 172.21.15.16:0/4073972219 <== osd.5 v1:172.21.15.183:6804/3356204259 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f5a8c07cd70 con 0x7f5a8c0790c0 2024-03-26T04:18:04.904 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.904+0000 7f5aa2d28700 1 -- 172.21.15.16:0/4073972219 --> v1:172.21.15.183:6804/3356204259 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5a80002c40 con 0x7f5a8c0790c0 2024-03-26T04:18:04.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.904+0000 7f5a99ffb700 1 -- 172.21.15.16:0/4073972219 <== osd.5 v1:172.21.15.183:6804/3356204259 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (unknown 0 0 3495208856) 0x7f5a80002c40 con 0x7f5a8c0790c0 2024-03-26T04:18:04.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.905+0000 7f5aa2d28700 1 -- 172.21.15.16:0/4073972219 >> v1:172.21.15.183:6804/3356204259 conn(0x7f5a8c0790c0 legacy=0x7f5a8c07b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:04.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.905+0000 7f5aa2d28700 1 -- 172.21.15.16:0/4073972219 >> v1:172.21.15.16:6800/3868004154 conn(0x7f5a8c06ea40 legacy=0x7f5a8c070f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:04.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.905+0000 7f5aa2d28700 1 -- 172.21.15.16:0/4073972219 >> v1:172.21.15.16:6790/0 conn(0x7f5a9c10ad10 legacy=0x7f5a9c1a1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:04.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.905+0000 7f5aa12c5700 1 -- 172.21.15.16:0/4073972219 reap_dead start 2024-03-26T04:18:04.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.905+0000 7f5aa2d28700 1 -- 172.21.15.16:0/4073972219 shutdown_connections 2024-03-26T04:18:04.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.905+0000 7f5aa2d28700 1 -- 172.21.15.16:0/4073972219 >> 172.21.15.16:0/4073972219 conn(0x7f5a9c0fd930 msgr2=0x7f5a9c111a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:18:04.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.906+0000 7f5aa2d28700 1 -- 172.21.15.16:0/4073972219 shutdown_connections 2024-03-26T04:18:04.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:04.906+0000 7f5aa2d28700 1 -- 172.21.15.16:0/4073972219 wait complete. 2024-03-26T04:18:05.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.601+0000 7fa52c442700 1 -- 172.21.15.16:0/2777188292 >> v1:172.21.15.16:6789/0 conn(0x7fa52410ae40 legacy=0x7fa52410d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:05.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.602+0000 7fa52c442700 1 -- 172.21.15.16:0/2777188292 shutdown_connections 2024-03-26T04:18:05.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.602+0000 7fa52c442700 1 -- 172.21.15.16:0/2777188292 >> 172.21.15.16:0/2777188292 conn(0x7fa5240fc820 msgr2=0x7fa5240fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:18:05.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.602+0000 7fa52c442700 1 -- 172.21.15.16:0/2777188292 shutdown_connections 2024-03-26T04:18:05.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.602+0000 7fa52c442700 1 -- 172.21.15.16:0/2777188292 wait complete. 2024-03-26T04:18:05.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.602+0000 7fa52c442700 1 Processor -- start 2024-03-26T04:18:05.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.602+0000 7fa52c442700 1 -- start start 2024-03-26T04:18:05.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.602+0000 7fa52c442700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa5241a8f70 con 0x7fa52410ae40 2024-03-26T04:18:05.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.602+0000 7fa52c442700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa5241aa150 con 0x7fa52410e950 2024-03-26T04:18:05.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.602+0000 7fa52c442700 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa5241ab350 con 0x7fa524103030 2024-03-26T04:18:05.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.603+0000 7fa5299dd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa52410ae40 0x7fa5241a4020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54792/0 (socket says 172.21.15.16:54792) 2024-03-26T04:18:05.603 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.603+0000 7fa5299dd700 1 -- 172.21.15.16:0/4045207999 learned_addr learned my addr 172.21.15.16:0/4045207999 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:18:05.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.603+0000 7fa51b7fe700 1 -- 172.21.15.16:0/4045207999 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2630129250 0 0) 0x7fa5241ab350 con 0x7fa524103030 2024-03-26T04:18:05.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.603+0000 7fa51b7fe700 1 -- 172.21.15.16:0/4045207999 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5100031f0 con 0x7fa524103030 2024-03-26T04:18:05.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.603+0000 7fa51b7fe700 1 -- 172.21.15.16:0/4045207999 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3806190735 0 0) 0x7fa5241a8f70 con 0x7fa52410ae40 2024-03-26T04:18:05.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.603+0000 7fa51b7fe700 1 -- 172.21.15.16:0/4045207999 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5241ab350 con 0x7fa52410ae40 2024-03-26T04:18:05.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.603+0000 7fa51b7fe700 1 -- 172.21.15.16:0/4045207999 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3211270562 0 0) 0x7fa5241aa150 con 0x7fa52410e950 2024-03-26T04:18:05.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.603+0000 7fa51b7fe700 1 -- 172.21.15.16:0/4045207999 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5241a8f70 con 0x7fa52410e950 2024-03-26T04:18:05.605 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.603+0000 7fa51b7fe700 1 -- 172.21.15.16:0/4045207999 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1748794711 0 0) 0x7fa5100031f0 con 0x7fa524103030 2024-03-26T04:18:05.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.603+0000 7fa51b7fe700 1 -- 172.21.15.16:0/4045207999 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5241aa150 con 0x7fa524103030 2024-03-26T04:18:05.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.603+0000 7fa51b7fe700 1 -- 172.21.15.16:0/4045207999 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 59296042 0 0) 0x7fa5241ab350 con 0x7fa52410ae40 2024-03-26T04:18:05.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.603+0000 7fa51b7fe700 1 -- 172.21.15.16:0/4045207999 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5100031f0 con 0x7fa52410ae40 2024-03-26T04:18:05.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.603+0000 7fa51b7fe700 1 -- 172.21.15.16:0/4045207999 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fa5200018d0 con 0x7fa524103030 2024-03-26T04:18:05.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.603+0000 7fa51b7fe700 1 -- 172.21.15.16:0/4045207999 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fa5140015d0 con 0x7fa52410ae40 2024-03-26T04:18:05.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.603+0000 7fa51b7fe700 1 -- 172.21.15.16:0/4045207999 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2883282069 0 0) 0x7fa5241a8f70 con 0x7fa52410e950 2024-03-26T04:18:05.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.603+0000 7fa51b7fe700 1 -- 172.21.15.16:0/4045207999 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5241ab350 con 0x7fa52410e950 2024-03-26T04:18:05.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.603+0000 7fa51b7fe700 1 -- 172.21.15.16:0/4045207999 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fa51c001af0 con 0x7fa52410e950 2024-03-26T04:18:05.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.603+0000 7fa51b7fe700 1 -- 172.21.15.16:0/4045207999 <== mon.2 v1:172.21.15.16:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2926163701 0 0) 0x7fa5241aa150 con 0x7fa524103030 2024-03-26T04:18:05.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.603+0000 7fa51b7fe700 1 -- 172.21.15.16:0/4045207999 >> v1:172.21.15.183:6789/0 conn(0x7fa52410e950 legacy=0x7fa5241a7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:05.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.603+0000 7fa51b7fe700 1 -- 172.21.15.16:0/4045207999 >> v1:172.21.15.16:6789/0 conn(0x7fa52410ae40 legacy=0x7fa5241a4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:05.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.603+0000 7fa51b7fe700 1 -- 172.21.15.16:0/4045207999 --> v1:172.21.15.16:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5241ac550 con 0x7fa524103030 2024-03-26T04:18:05.606 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.603+0000 7fa52c442700 1 -- 172.21.15.16:0/4045207999 --> v1:172.21.15.16:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5241ab540 con 0x7fa524103030 2024-03-26T04:18:05.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.604+0000 7fa52c442700 1 -- 172.21.15.16:0/4045207999 --> v1:172.21.15.16:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5241ab9c0 con 0x7fa524103030 2024-03-26T04:18:05.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.604+0000 7fa51b7fe700 1 -- 172.21.15.16:0/4045207999 <== mon.2 v1:172.21.15.16:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7fa5200010b0 con 0x7fa524103030 2024-03-26T04:18:05.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.604+0000 7fa51b7fe700 1 -- 172.21.15.16:0/4045207999 <== mon.2 v1:172.21.15.16:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fa520004920 con 0x7fa524103030 2024-03-26T04:18:05.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.604+0000 7fa52c442700 1 -- 172.21.15.16:0/4045207999 --> v1:172.21.15.16:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa508000fb0 con 0x7fa524103030 2024-03-26T04:18:05.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.605+0000 7fa51b7fe700 1 -- 172.21.15.16:0/4045207999 <== mon.2 v1:172.21.15.16:6790/0 7 ==== mgrmap(e 22) v1 ==== 91420+0+0 (unknown 3095259778 0 0) 0x7fa52001b090 con 0x7fa524103030 2024-03-26T04:18:05.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.605+0000 7fa51b7fe700 1 -- 172.21.15.16:0/4045207999 <== mon.2 v1:172.21.15.16:6790/0 8 ==== osd_map(70..70 src has 1..70) v4 ==== 6172+0+0 (unknown 1260882646 0 0) 0x7fa5200876b0 con 0x7fa524103030 2024-03-26T04:18:05.607 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.606+0000 7fa51b7fe700 1 -- 172.21.15.16:0/4045207999 --> v1:172.21.15.183:6800/879252532 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa51007cfc0 con 0x7fa510079310 2024-03-26T04:18:05.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.606+0000 7fa51b7fe700 1 -- 172.21.15.16:0/4045207999 <== mon.2 v1:172.21.15.16:6790/0 9 ==== mon_get_version_reply(handle=1 version=70) v2 ==== 24+0+0 (unknown 2350636211 0 0) 0x7fa5200879f0 con 0x7fa524103030 2024-03-26T04:18:05.608 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.607+0000 7fa51b7fe700 1 -- 172.21.15.16:0/4045207999 <== osd.4 v1:172.21.15.183:6800/879252532 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fa51007cfc0 con 0x7fa510079310 2024-03-26T04:18:05.624 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.623+0000 7fa52c442700 1 -- 172.21.15.16:0/4045207999 --> v1:172.21.15.183:6800/879252532 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa508002c70 con 0x7fa510079310 2024-03-26T04:18:05.625 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.624+0000 7fa51b7fe700 1 -- 172.21.15.16:0/4045207999 <== osd.4 v1:172.21.15.183:6800/879252532 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (unknown 0 0 804844592) 0x7fa508002c70 con 0x7fa510079310 2024-03-26T04:18:05.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.624+0000 7fa52c442700 1 -- 172.21.15.16:0/4045207999 >> v1:172.21.15.183:6800/879252532 conn(0x7fa510079310 legacy=0x7fa51007b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:05.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.625+0000 7fa52c442700 1 -- 172.21.15.16:0/4045207999 >> v1:172.21.15.16:6800/3868004154 conn(0x7fa51006eb90 legacy=0x7fa510071050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:05.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.625+0000 7fa52c442700 1 -- 172.21.15.16:0/4045207999 >> v1:172.21.15.16:6790/0 conn(0x7fa524103030 legacy=0x7fa52410a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:05.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.625+0000 7fa52a9df700 1 -- 172.21.15.16:0/4045207999 reap_dead start 2024-03-26T04:18:05.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.625+0000 7fa52c442700 1 -- 172.21.15.16:0/4045207999 shutdown_connections 2024-03-26T04:18:05.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.625+0000 7fa52c442700 1 -- 172.21.15.16:0/4045207999 >> 172.21.15.16:0/4045207999 conn(0x7fa5240fc820 msgr2=0x7fa5240fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:18:05.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.625+0000 7fa52c442700 1 -- 172.21.15.16:0/4045207999 shutdown_connections 2024-03-26T04:18:05.626 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:05.625+0000 7fa52c442700 1 -- 172.21.15.16:0/4045207999 wait complete. 2024-03-26T04:18:06.100 INFO:teuthology.orchestra.run.smithi016.stdout:163208757271 2024-03-26T04:18:06.100 DEBUG:teuthology.orchestra.run.smithi016:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-26T04:18:06.367 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:06 smithi016 ceph-mon[29101]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T04:18:06.367 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:06 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:18:06.367 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:06 smithi016 ceph-mon[37215]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T04:18:06.367 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:06 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:18:06.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:06 smithi183 ceph-mon[33745]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T04:18:06.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:06 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:18:06.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.856+0000 7f73f7a73700 1 -- 172.21.15.16:0/1601755229 >> v1:172.21.15.16:6789/0 conn(0x7f73f0100f70 legacy=0x7f73f0101350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:06.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.856+0000 7f73f7a73700 1 -- 172.21.15.16:0/1601755229 shutdown_connections 2024-03-26T04:18:06.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.856+0000 7f73f7a73700 1 -- 172.21.15.16:0/1601755229 >> 172.21.15.16:0/1601755229 conn(0x7f73f00fc820 msgr2=0x7f73f00fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:18:06.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.856+0000 7f73f7a73700 1 -- 172.21.15.16:0/1601755229 shutdown_connections 2024-03-26T04:18:06.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.856+0000 7f73f7a73700 1 -- 172.21.15.16:0/1601755229 wait complete. 2024-03-26T04:18:06.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.856+0000 7f73f7a73700 1 Processor -- start 2024-03-26T04:18:06.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.857+0000 7f73f7a73700 1 -- start start 2024-03-26T04:18:06.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.857+0000 7f73f7a73700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f73f01a8f60 con 0x7f73f0109c00 2024-03-26T04:18:06.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.857+0000 7f73f7a73700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f73f01aa140 con 0x7f73f0100f70 2024-03-26T04:18:06.859 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.857+0000 7f73f7a73700 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f73f01ab340 con 0x7f73f010d7a0 2024-03-26T04:18:06.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.857+0000 7f73f6010700 1 --1- >> v1:172.21.15.16:6790/0 conn(0x7f73f010d7a0 0x7f73f01a7640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6790/0 says I am v1:172.21.15.16:39072/0 (socket says 172.21.15.16:39072) 2024-03-26T04:18:06.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.857+0000 7f73f6010700 1 -- 172.21.15.16:0/1185506320 learned_addr learned my addr 172.21.15.16:0/1185506320 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:18:06.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.857+0000 7f73eeffd700 1 -- 172.21.15.16:0/1185506320 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2176825507 0 0) 0x7f73f01a8f60 con 0x7f73f0109c00 2024-03-26T04:18:06.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.857+0000 7f73eeffd700 1 -- 172.21.15.16:0/1185506320 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73d80031f0 con 0x7f73f0109c00 2024-03-26T04:18:06.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.857+0000 7f73eeffd700 1 -- 172.21.15.16:0/1185506320 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 641323848 0 0) 0x7f73f01ab340 con 0x7f73f010d7a0 2024-03-26T04:18:06.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.857+0000 7f73eeffd700 1 -- 172.21.15.16:0/1185506320 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73f01a8f60 con 0x7f73f010d7a0 2024-03-26T04:18:06.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.857+0000 7f73eeffd700 1 -- 172.21.15.16:0/1185506320 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 604701134 0 0) 0x7f73d80031f0 con 0x7f73f0109c00 2024-03-26T04:18:06.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.857+0000 7f73eeffd700 1 -- 172.21.15.16:0/1185506320 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73f01ab340 con 0x7f73f0109c00 2024-03-26T04:18:06.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.857+0000 7f73eeffd700 1 -- 172.21.15.16:0/1185506320 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2675076951 0 0) 0x7f73f01a8f60 con 0x7f73f010d7a0 2024-03-26T04:18:06.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.857+0000 7f73eeffd700 1 -- 172.21.15.16:0/1185506320 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73d80031f0 con 0x7f73f010d7a0 2024-03-26T04:18:06.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.857+0000 7f73eeffd700 1 -- 172.21.15.16:0/1185506320 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 910557019 0 0) 0x7f73f01aa140 con 0x7f73f0100f70 2024-03-26T04:18:06.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.857+0000 7f73eeffd700 1 -- 172.21.15.16:0/1185506320 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73f01a8f60 con 0x7f73f0100f70 2024-03-26T04:18:06.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.858+0000 7f73eeffd700 1 -- 172.21.15.16:0/1185506320 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f73e40019f0 con 0x7f73f0109c00 2024-03-26T04:18:06.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.858+0000 7f73eeffd700 1 -- 172.21.15.16:0/1185506320 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f73e8001870 con 0x7f73f010d7a0 2024-03-26T04:18:06.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.858+0000 7f73eeffd700 1 -- 172.21.15.16:0/1185506320 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 368954332 0 0) 0x7f73f01ab340 con 0x7f73f0109c00 2024-03-26T04:18:06.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.858+0000 7f73eeffd700 1 -- 172.21.15.16:0/1185506320 >> v1:172.21.15.16:6790/0 conn(0x7f73f010d7a0 legacy=0x7f73f01a7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:06.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.858+0000 7f73eeffd700 1 -- 172.21.15.16:0/1185506320 >> v1:172.21.15.183:6789/0 conn(0x7f73f0100f70 legacy=0x7f73f0106270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:06.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.858+0000 7f73eeffd700 1 -- 172.21.15.16:0/1185506320 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73f01ac540 con 0x7f73f0109c00 2024-03-26T04:18:06.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.858+0000 7f73f7a73700 1 -- 172.21.15.16:0/1185506320 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73f01ab530 con 0x7f73f0109c00 2024-03-26T04:18:06.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.858+0000 7f73f7a73700 1 -- 172.21.15.16:0/1185506320 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f73f01ab9b0 con 0x7f73f0109c00 2024-03-26T04:18:06.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.858+0000 7f73eeffd700 1 -- 172.21.15.16:0/1185506320 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f73e4003b10 con 0x7f73f0109c00 2024-03-26T04:18:06.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.858+0000 7f73eeffd700 1 -- 172.21.15.16:0/1185506320 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f73e40048e0 con 0x7f73f0109c00 2024-03-26T04:18:06.861 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.859+0000 7f73f7a73700 1 -- 172.21.15.16:0/1185506320 --> v1:172.21.15.16:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f73dc000f80 con 0x7f73f0109c00 2024-03-26T04:18:06.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.859+0000 7f73eeffd700 1 -- 172.21.15.16:0/1185506320 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 22) v1 ==== 91420+0+0 (unknown 3095259778 0 0) 0x7f73e4003680 con 0x7f73f0109c00 2024-03-26T04:18:06.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.860+0000 7f73eeffd700 1 -- 172.21.15.16:0/1185506320 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(70..70 src has 1..70) v4 ==== 6172+0+0 (unknown 1260882646 0 0) 0x7f73e4087580 con 0x7f73f0109c00 2024-03-26T04:18:06.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.860+0000 7f73eeffd700 1 -- 172.21.15.16:0/1185506320 --> v1:172.21.15.183:6808/777758685 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f73d807ce00 con 0x7f73d8079150 2024-03-26T04:18:06.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.860+0000 7f73eeffd700 1 -- 172.21.15.16:0/1185506320 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_get_version_reply(handle=1 version=70) v2 ==== 24+0+0 (unknown 2350636211 0 0) 0x7f73e4087990 con 0x7f73f0109c00 2024-03-26T04:18:06.862 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.861+0000 7f73eeffd700 1 -- 172.21.15.16:0/1185506320 <== osd.6 v1:172.21.15.183:6808/777758685 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f73d807ce00 con 0x7f73d8079150 2024-03-26T04:18:06.878 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.878+0000 7f73f7a73700 1 -- 172.21.15.16:0/1185506320 --> v1:172.21.15.183:6808/777758685 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f73dc002c40 con 0x7f73d8079150 2024-03-26T04:18:06.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.878+0000 7f73eeffd700 1 -- 172.21.15.16:0/1185506320 <== osd.6 v1:172.21.15.183:6808/777758685 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (unknown 0 0 1795465705) 0x7f73dc002c40 con 0x7f73d8079150 2024-03-26T04:18:06.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.879+0000 7f73f7a73700 1 -- 172.21.15.16:0/1185506320 >> v1:172.21.15.183:6808/777758685 conn(0x7f73d8079150 legacy=0x7f73d807b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:06.879 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.879+0000 7f73f7a73700 1 -- 172.21.15.16:0/1185506320 >> v1:172.21.15.16:6800/3868004154 conn(0x7f73d806ea50 legacy=0x7f73d8070f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:06.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.879+0000 7f73f7a73700 1 -- 172.21.15.16:0/1185506320 >> v1:172.21.15.16:6789/0 conn(0x7f73f0109c00 legacy=0x7f73f01a3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:06.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.879+0000 7f73f6010700 1 -- 172.21.15.16:0/1185506320 reap_dead start 2024-03-26T04:18:06.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.879+0000 7f73f7a73700 1 -- 172.21.15.16:0/1185506320 shutdown_connections 2024-03-26T04:18:06.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.879+0000 7f73f7a73700 1 -- 172.21.15.16:0/1185506320 >> 172.21.15.16:0/1185506320 conn(0x7f73f00fc820 msgr2=0x7f73f006a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:18:06.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.879+0000 7f73f7a73700 1 -- 172.21.15.16:0/1185506320 shutdown_connections 2024-03-26T04:18:06.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:06.879+0000 7f73f7a73700 1 -- 172.21.15.16:0/1185506320 wait complete. 2024-03-26T04:18:07.328 INFO:teuthology.orchestra.run.smithi016.stdout:128849018909 2024-03-26T04:18:07.328 DEBUG:teuthology.orchestra.run.smithi016:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-26T04:18:07.944 INFO:teuthology.orchestra.run.smithi016.stdout:197568495636 2024-03-26T04:18:07.945 DEBUG:teuthology.orchestra.run.smithi016:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-26T04:18:08.514 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:08 smithi016 ceph-mon[37215]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T04:18:08.515 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:08 smithi016 ceph-mon[29101]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T04:18:08.534 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:18:08.534 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:18:08.537 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:18:08.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:08 smithi183 ceph-mon[33745]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-26T04:18:08.564 INFO:teuthology.orchestra.run.smithi016.stdout:107374182435 2024-03-26T04:18:08.565 DEBUG:teuthology.orchestra.run.smithi016:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-26T04:18:08.601 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:18:08.603 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:18:08.604 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:18:08.840 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:18:08 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:18:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:18:10.587 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:10 smithi016 ceph-mon[29101]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:18:10.588 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:10 smithi016 ceph-mon[37215]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:18:10.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:10 smithi183 ceph-mon[33745]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:18:12.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:18:11 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:18:12.534 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:12 smithi016 ceph-mon[37215]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:18:12.534 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:12 smithi016 ceph-mon[29101]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:18:12.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:12.642+0000 7f5af3961700 1 -- 172.21.15.16:0/2265931295 >> v1:172.21.15.16:6790/0 conn(0x7f5aec10ae80 legacy=0x7f5aec10d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:12.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:12.642+0000 7f5af3961700 1 -- 172.21.15.16:0/2265931295 shutdown_connections 2024-03-26T04:18:12.642 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:12.642+0000 7f5af3961700 1 -- 172.21.15.16:0/2265931295 >> 172.21.15.16:0/2265931295 conn(0x7f5aec0fc820 msgr2=0x7f5aec0fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:18:12.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:12.642+0000 7f5af3961700 1 -- 172.21.15.16:0/2265931295 shutdown_connections 2024-03-26T04:18:12.643 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:12.642+0000 7f5af3961700 1 -- 172.21.15.16:0/2265931295 wait complete. 2024-03-26T04:18:12.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:12.642+0000 7f5af3961700 1 Processor -- start 2024-03-26T04:18:12.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:12.643+0000 7f5af3961700 1 -- start start 2024-03-26T04:18:12.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:12.643+0000 7f5af3961700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5aec1a6c80 con 0x7f5aec10e990 2024-03-26T04:18:12.644 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:12.643+0000 7f5af3961700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5aec1a7e60 con 0x7f5aec103070 2024-03-26T04:18:12.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:12.643+0000 7f5af3961700 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5aec1a9060 con 0x7f5aec10ae80 2024-03-26T04:18:12.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:12.643+0000 7f5af0efc700 1 --1- >> v1:172.21.15.16:6790/0 conn(0x7f5aec10ae80 0x7f5aec1a1c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6790/0 says I am v1:172.21.15.16:39086/0 (socket says 172.21.15.16:39086) 2024-03-26T04:18:12.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:12.643+0000 7f5af1efe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5aec10e990 0x7f5aec1a5340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54820/0 (socket says 172.21.15.16:54820) 2024-03-26T04:18:12.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:12.643+0000 7f5af0efc700 1 -- 172.21.15.16:0/3547233630 learned_addr learned my addr 172.21.15.16:0/3547233630 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:18:12.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:12.643+0000 7f5ae27fc700 1 -- 172.21.15.16:0/3547233630 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 767938666 0 0) 0x7f5aec1a6c80 con 0x7f5aec10e990 2024-03-26T04:18:12.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:12.643+0000 7f5ae27fc700 1 -- 172.21.15.16:0/3547233630 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ad80035b0 con 0x7f5aec10e990 2024-03-26T04:18:12.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:12.643+0000 7f5ae27fc700 1 -- 172.21.15.16:0/3547233630 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1289898395 0 0) 0x7f5aec1a9060 con 0x7f5aec10ae80 2024-03-26T04:18:12.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:12.643+0000 7f5ae27fc700 1 -- 172.21.15.16:0/3547233630 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5aec1a6c80 con 0x7f5aec10ae80 2024-03-26T04:18:12.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:12.643+0000 7f5ae27fc700 1 -- 172.21.15.16:0/3547233630 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2836772623 0 0) 0x7f5aec1a7e60 con 0x7f5aec103070 2024-03-26T04:18:12.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:12.643+0000 7f5ae27fc700 1 -- 172.21.15.16:0/3547233630 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5aec1a9060 con 0x7f5aec103070 2024-03-26T04:18:12.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:12.643+0000 7f5ae27fc700 1 -- 172.21.15.16:0/3547233630 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 727304293 0 0) 0x7f5ad80035b0 con 0x7f5aec10e990 2024-03-26T04:18:12.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:12.643+0000 7f5ae27fc700 1 -- 172.21.15.16:0/3547233630 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5aec1a7e60 con 0x7f5aec10e990 2024-03-26T04:18:12.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:12.643+0000 7f5ae27fc700 1 -- 172.21.15.16:0/3547233630 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f5ae4001bb0 con 0x7f5aec10e990 2024-03-26T04:18:12.645 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:12.643+0000 7f5ae27fc700 1 -- 172.21.15.16:0/3547233630 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3953798430 0 0) 0x7f5aec1a6c80 con 0x7f5aec10ae80 2024-03-26T04:18:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:12.644+0000 7f5ae27fc700 1 -- 172.21.15.16:0/3547233630 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ad80035b0 con 0x7f5aec10ae80 2024-03-26T04:18:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:12.644+0000 7f5ae27fc700 1 -- 172.21.15.16:0/3547233630 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f5ae80045d0 con 0x7f5aec10ae80 2024-03-26T04:18:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:12.644+0000 7f5ae27fc700 1 -- 172.21.15.16:0/3547233630 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 136328922 0 0) 0x7f5aec1a7e60 con 0x7f5aec10e990 2024-03-26T04:18:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:12.644+0000 7f5ae27fc700 1 -- 172.21.15.16:0/3547233630 >> v1:172.21.15.16:6790/0 conn(0x7f5aec10ae80 legacy=0x7f5aec1a1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:12.644+0000 7f5ae27fc700 1 -- 172.21.15.16:0/3547233630 >> v1:172.21.15.183:6789/0 conn(0x7f5aec103070 legacy=0x7f5aec10a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:12.644+0000 7f5ae27fc700 1 -- 172.21.15.16:0/3547233630 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aec1aa260 con 0x7f5aec10e990 2024-03-26T04:18:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:12.644+0000 7f5af3961700 1 -- 172.21.15.16:0/3547233630 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aec1a9250 con 0x7f5aec10e990 2024-03-26T04:18:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:12.644+0000 7f5af3961700 1 -- 172.21.15.16:0/3547233630 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aec1a96d0 con 0x7f5aec10e990 2024-03-26T04:18:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:12.644+0000 7f5ae27fc700 1 -- 172.21.15.16:0/3547233630 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f5ae4003eb0 con 0x7f5aec10e990 2024-03-26T04:18:12.646 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:12.644+0000 7f5ae27fc700 1 -- 172.21.15.16:0/3547233630 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f5ae4005f40 con 0x7f5aec10e990 2024-03-26T04:18:12.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:12.645+0000 7f5af3961700 1 -- 172.21.15.16:0/3547233630 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ad0005120 con 0x7f5aec10e990 2024-03-26T04:18:12.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:12.645+0000 7f5ae27fc700 1 -- 172.21.15.16:0/3547233630 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 22) v1 ==== 91420+0+0 (unknown 3095259778 0 0) 0x7f5ae4003a20 con 0x7f5aec10e990 2024-03-26T04:18:12.647 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:12.646+0000 7f5ae27fc700 1 -- 172.21.15.16:0/3547233630 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(70..70 src has 1..70) v4 ==== 6172+0+0 (unknown 1260882646 0 0) 0x7f5ae4088b80 con 0x7f5aec10e990 2024-03-26T04:18:12.650 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:12.650+0000 7f5ae27fc700 1 -- 172.21.15.16:0/3547233630 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f5ae4055fc0 con 0x7f5aec10e990 2024-03-26T04:18:12.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:12 smithi183 ceph-mon[33745]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:18:12.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:12.859+0000 7f5af3961700 1 -- 172.21.15.16:0/3547233630 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f5ad00053d0 con 0x7f5aec10e990 2024-03-26T04:18:12.860 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:12.859+0000 7f5ae27fc700 1 -- 172.21.15.16:0/3547233630 <== mon.0 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (unknown 92182286 0 3693870858) 0x7f5ae4054330 con 0x7f5aec10e990 2024-03-26T04:18:12.860 INFO:teuthology.orchestra.run.smithi016.stdout:77309411369 2024-03-26T04:18:12.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:12.862+0000 7f5af3961700 1 -- 172.21.15.16:0/3547233630 >> v1:172.21.15.16:6800/3868004154 conn(0x7f5ad806e7d0 legacy=0x7f5ad8070c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:12.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:12.862+0000 7f5af3961700 1 -- 172.21.15.16:0/3547233630 >> v1:172.21.15.16:6789/0 conn(0x7f5aec10e990 legacy=0x7f5aec1a5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:12.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:12.862+0000 7f5af3961700 1 -- 172.21.15.16:0/3547233630 shutdown_connections 2024-03-26T04:18:12.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:12.862+0000 7f5af3961700 1 -- 172.21.15.16:0/3547233630 >> 172.21.15.16:0/3547233630 conn(0x7f5aec0fc820 msgr2=0x7f5aec0fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:18:12.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:12.862+0000 7f5af3961700 1 -- 172.21.15.16:0/3547233630 shutdown_connections 2024-03-26T04:18:12.863 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:12.862+0000 7f5af3961700 1 -- 172.21.15.16:0/3547233630 wait complete. 2024-03-26T04:18:13.526 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:13 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:18:13.526 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:13 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3547233630' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T04:18:13.527 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:13 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:18:13.527 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:13 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3547233630' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T04:18:13.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:13 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:18:13.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:13 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3547233630' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-26T04:18:14.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:14 smithi016 ceph-mon[37215]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:18:14.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:14 smithi016 ceph-mon[29101]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:18:14.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.232+0000 7f65fe136700 1 -- 172.21.15.16:0/344880867 >> v1:172.21.15.16:6790/0 conn(0x7f65f810d7b0 legacy=0x7f65f810fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:14.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.232+0000 7f65fe136700 1 -- 172.21.15.16:0/344880867 shutdown_connections 2024-03-26T04:18:14.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.232+0000 7f65fe136700 1 -- 172.21.15.16:0/344880867 >> 172.21.15.16:0/344880867 conn(0x7f65f80fc830 msgr2=0x7f65f80fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:18:14.656 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.232+0000 7f65fe136700 1 -- 172.21.15.16:0/344880867 shutdown_connections 2024-03-26T04:18:14.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.232+0000 7f65fe136700 1 -- 172.21.15.16:0/344880867 wait complete. 2024-03-26T04:18:14.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.232+0000 7f65fe136700 1 Processor -- start 2024-03-26T04:18:14.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.232+0000 7f65fe136700 1 -- start start 2024-03-26T04:18:14.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.233+0000 7f65fe136700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f65f81a6dd0 con 0x7f65f8100f80 2024-03-26T04:18:14.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.233+0000 7f65fe136700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f65f81a7fb0 con 0x7f65f810d7b0 2024-03-26T04:18:14.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.233+0000 7f65fe136700 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f65f81a91b0 con 0x7f65f8109c10 2024-03-26T04:18:14.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.233+0000 7f65f6ffd700 1 --1- >> v1:172.21.15.16:6790/0 conn(0x7f65f8109c10 0x7f65f81a1b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6790/0 says I am v1:172.21.15.16:39116/0 (socket says 172.21.15.16:39116) 2024-03-26T04:18:14.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.233+0000 7f65f6ffd700 1 -- 172.21.15.16:0/1778093924 learned_addr learned my addr 172.21.15.16:0/1778093924 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:18:14.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.233+0000 7f65f77fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f65f8100f80 0x7f65f8106280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54826/0 (socket says 172.21.15.16:54826) 2024-03-26T04:18:14.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.233+0000 7f65f4ff9700 1 -- 172.21.15.16:0/1778093924 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 218481556 0 0) 0x7f65f81a91b0 con 0x7f65f8109c10 2024-03-26T04:18:14.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.233+0000 7f65f4ff9700 1 -- 172.21.15.16:0/1778093924 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65e00035b0 con 0x7f65f8109c10 2024-03-26T04:18:14.657 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.233+0000 7f65f4ff9700 1 -- 172.21.15.16:0/1778093924 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1647896149 0 0) 0x7f65f81a6dd0 con 0x7f65f8100f80 2024-03-26T04:18:14.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.233+0000 7f65f4ff9700 1 -- 172.21.15.16:0/1778093924 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65f81a91b0 con 0x7f65f8100f80 2024-03-26T04:18:14.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.233+0000 7f65f4ff9700 1 -- 172.21.15.16:0/1778093924 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 557257648 0 0) 0x7f65f81a7fb0 con 0x7f65f810d7b0 2024-03-26T04:18:14.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.233+0000 7f65f4ff9700 1 -- 172.21.15.16:0/1778093924 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65f81a6dd0 con 0x7f65f810d7b0 2024-03-26T04:18:14.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.233+0000 7f65f4ff9700 1 -- 172.21.15.16:0/1778093924 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4029655885 0 0) 0x7f65e00035b0 con 0x7f65f8109c10 2024-03-26T04:18:14.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.233+0000 7f65f4ff9700 1 -- 172.21.15.16:0/1778093924 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65f81a7fb0 con 0x7f65f8109c10 2024-03-26T04:18:14.658 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.233+0000 7f65f4ff9700 1 -- 172.21.15.16:0/1778093924 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f65ec0018f0 con 0x7f65f8109c10 2024-03-26T04:18:14.658 INFO:teuthology.orchestra.run.smithi016.stdout:34359738419 2024-03-26T04:18:14.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.233+0000 7f65f4ff9700 1 -- 172.21.15.16:0/1778093924 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2028046215 0 0) 0x7f65f81a91b0 con 0x7f65f8100f80 2024-03-26T04:18:14.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.233+0000 7f65f4ff9700 1 -- 172.21.15.16:0/1778093924 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65e00035b0 con 0x7f65f8100f80 2024-03-26T04:18:14.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.233+0000 7f65f4ff9700 1 -- 172.21.15.16:0/1778093924 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f65e80018a0 con 0x7f65f8100f80 2024-03-26T04:18:14.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.233+0000 7f65f4ff9700 1 -- 172.21.15.16:0/1778093924 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 333933342 0 0) 0x7f65f81a6dd0 con 0x7f65f810d7b0 2024-03-26T04:18:14.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.233+0000 7f65f4ff9700 1 -- 172.21.15.16:0/1778093924 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f65f81a91b0 con 0x7f65f810d7b0 2024-03-26T04:18:14.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.233+0000 7f65f4ff9700 1 -- 172.21.15.16:0/1778093924 <== mon.2 v1:172.21.15.16:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1576891730 0 0) 0x7f65f81a7fb0 con 0x7f65f8109c10 2024-03-26T04:18:14.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.233+0000 7f65f4ff9700 1 -- 172.21.15.16:0/1778093924 >> v1:172.21.15.183:6789/0 conn(0x7f65f810d7b0 legacy=0x7f65f81a5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:14.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.233+0000 7f65f4ff9700 1 -- 172.21.15.16:0/1778093924 >> v1:172.21.15.16:6789/0 conn(0x7f65f8100f80 legacy=0x7f65f8106280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:14.659 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.233+0000 7f65f4ff9700 1 -- 172.21.15.16:0/1778093924 --> v1:172.21.15.16:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f81aa3b0 con 0x7f65f8109c10 2024-03-26T04:18:14.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.233+0000 7f65fe136700 1 -- 172.21.15.16:0/1778093924 --> v1:172.21.15.16:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65f81a6fc0 con 0x7f65f8109c10 2024-03-26T04:18:14.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.234+0000 7f65fe136700 1 -- 172.21.15.16:0/1778093924 --> v1:172.21.15.16:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f65f81a7440 con 0x7f65f8109c10 2024-03-26T04:18:14.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.234+0000 7f65f4ff9700 1 -- 172.21.15.16:0/1778093924 <== mon.2 v1:172.21.15.16:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f65ec003b30 con 0x7f65f8109c10 2024-03-26T04:18:14.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.234+0000 7f65f4ff9700 1 -- 172.21.15.16:0/1778093924 <== mon.2 v1:172.21.15.16:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f65ec005ca0 con 0x7f65f8109c10 2024-03-26T04:18:14.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.234+0000 7f65fe136700 1 -- 172.21.15.16:0/1778093924 --> v1:172.21.15.16:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65e4005120 con 0x7f65f8109c10 2024-03-26T04:18:14.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.235+0000 7f65f4ff9700 1 -- 172.21.15.16:0/1778093924 <== mon.2 v1:172.21.15.16:6790/0 7 ==== mgrmap(e 22) v1 ==== 91420+0+0 (unknown 3095259778 0 0) 0x7f65ec0036a0 con 0x7f65f8109c10 2024-03-26T04:18:14.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.236+0000 7f65f4ff9700 1 -- 172.21.15.16:0/1778093924 <== mon.2 v1:172.21.15.16:6790/0 8 ==== osd_map(70..70 src has 1..70) v4 ==== 6172+0+0 (unknown 1260882646 0 0) 0x7f65ec088920 con 0x7f65f8109c10 2024-03-26T04:18:14.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.240+0000 7f65f4ff9700 1 -- 172.21.15.16:0/1778093924 <== mon.2 v1:172.21.15.16:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f65ec055d60 con 0x7f65f8109c10 2024-03-26T04:18:14.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.455+0000 7f65fe136700 1 -- 172.21.15.16:0/1778093924 --> v1:172.21.15.16:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f65e40053d0 con 0x7f65f8109c10 2024-03-26T04:18:14.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.455+0000 7f65f4ff9700 1 -- 172.21.15.16:0/1778093924 <== mon.2 v1:172.21.15.16:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (unknown 574334944 0 2087499782) 0x7f65ec0540d0 con 0x7f65f8109c10 2024-03-26T04:18:14.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.457+0000 7f65fe136700 1 -- 172.21.15.16:0/1778093924 >> v1:172.21.15.16:6800/3868004154 conn(0x7f65e006eae0 legacy=0x7f65e0070fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:14.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.457+0000 7f65fe136700 1 -- 172.21.15.16:0/1778093924 >> v1:172.21.15.16:6790/0 conn(0x7f65f8109c10 legacy=0x7f65f81a1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:14.660 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.457+0000 7f65fe136700 1 -- 172.21.15.16:0/1778093924 shutdown_connections 2024-03-26T04:18:14.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.457+0000 7f65fe136700 1 -- 172.21.15.16:0/1778093924 >> 172.21.15.16:0/1778093924 conn(0x7f65f80fc830 msgr2=0x7f65f8110420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:18:14.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.458+0000 7f65fe136700 1 -- 172.21.15.16:0/1778093924 shutdown_connections 2024-03-26T04:18:14.661 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:14.458+0000 7f65fe136700 1 -- 172.21.15.16:0/1778093924 wait complete. 2024-03-26T04:18:14.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:14 smithi183 ceph-mon[33745]: pgmap v14: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:18:15.286 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738416 got 34359738419 for osd.0 2024-03-26T04:18:15.286 DEBUG:teuthology.parallel:result is None 2024-03-26T04:18:15.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:15 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1778093924' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T04:18:15.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:15 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1778093924' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T04:18:15.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:15 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1778093924' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-26T04:18:16.450 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:16 smithi016 ceph-mon[29101]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:18:16.450 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:16 smithi016 ceph-mon[37215]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:18:16.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:16 smithi183 ceph-mon[33745]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:18:17.485 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411367 got 77309411369 for osd.2 2024-03-26T04:18:17.486 DEBUG:teuthology.parallel:result is None 2024-03-26T04:18:18.497 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:18:18.600 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:18 smithi016 ceph-mon[37215]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:18:18.600 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:18 smithi016 ceph-mon[29101]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:18:18.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:18 smithi183 ceph-mon[33745]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:18:18.903 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:18:18 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:18:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:18:18.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:18.925+0000 7f8b72d70700 1 -- 172.21.15.16:0/3112541535 >> v1:172.21.15.16:6789/0 conn(0x7f8b6c10aea0 legacy=0x7f8b6c10d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:18.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:18.925+0000 7f8b72d70700 1 -- 172.21.15.16:0/3112541535 shutdown_connections 2024-03-26T04:18:18.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:18.925+0000 7f8b72d70700 1 -- 172.21.15.16:0/3112541535 >> 172.21.15.16:0/3112541535 conn(0x7f8b6c0fc820 msgr2=0x7f8b6c0fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:18:18.926 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:18.925+0000 7f8b72d70700 1 -- 172.21.15.16:0/3112541535 shutdown_connections 2024-03-26T04:18:18.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:18.925+0000 7f8b72d70700 1 -- 172.21.15.16:0/3112541535 wait complete. 2024-03-26T04:18:18.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:18.926+0000 7f8b72d70700 1 Processor -- start 2024-03-26T04:18:18.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:18.926+0000 7f8b72d70700 1 -- start start 2024-03-26T04:18:18.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:18.926+0000 7f8b72d70700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8b6c1a6ca0 con 0x7f8b6c103090 2024-03-26T04:18:18.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:18.926+0000 7f8b72d70700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8b6c1a7e80 con 0x7f8b6c10e9b0 2024-03-26T04:18:18.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:18.926+0000 7f8b72d70700 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f8b6c1a9080 con 0x7f8b6c10aea0 2024-03-26T04:18:18.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:18.926+0000 7f8b70b0c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8b6c103090 0x7f8b6c10a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44464/0 (socket says 172.21.15.16:44464) 2024-03-26T04:18:18.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:18.926+0000 7f8b70b0c700 1 -- 172.21.15.16:0/664459661 learned_addr learned my addr 172.21.15.16:0/664459661 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:18:18.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:18.926+0000 7f8b6bfff700 1 --1- 172.21.15.16:0/664459661 >> v1:172.21.15.16:6790/0 conn(0x7f8b6c10aea0 0x7f8b6c1a1c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6790/0 says I am v1:172.21.15.16:39592/0 (socket says 172.21.15.16:39592) 2024-03-26T04:18:18.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:18.926+0000 7f8b69ffb700 1 -- 172.21.15.16:0/664459661 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1096693606 0 0) 0x7f8b6c1a6ca0 con 0x7f8b6c103090 2024-03-26T04:18:18.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:18.926+0000 7f8b69ffb700 1 -- 172.21.15.16:0/664459661 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b580031f0 con 0x7f8b6c103090 2024-03-26T04:18:18.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:18.926+0000 7f8b69ffb700 1 -- 172.21.15.16:0/664459661 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3283464313 0 0) 0x7f8b6c1a9080 con 0x7f8b6c10aea0 2024-03-26T04:18:18.927 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:18.926+0000 7f8b69ffb700 1 -- 172.21.15.16:0/664459661 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b6c1a6ca0 con 0x7f8b6c10aea0 2024-03-26T04:18:18.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:18.926+0000 7f8b69ffb700 1 -- 172.21.15.16:0/664459661 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 225615987 0 0) 0x7f8b6c1a7e80 con 0x7f8b6c10e9b0 2024-03-26T04:18:18.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:18.926+0000 7f8b69ffb700 1 -- 172.21.15.16:0/664459661 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b6c1a9080 con 0x7f8b6c10e9b0 2024-03-26T04:18:18.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:18.927+0000 7f8b69ffb700 1 -- 172.21.15.16:0/664459661 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4013950217 0 0) 0x7f8b580031f0 con 0x7f8b6c103090 2024-03-26T04:18:18.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:18.927+0000 7f8b69ffb700 1 -- 172.21.15.16:0/664459661 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b6c1a7e80 con 0x7f8b6c103090 2024-03-26T04:18:18.929 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:18.927+0000 7f8b69ffb700 1 -- 172.21.15.16:0/664459661 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f8b5c0018d0 con 0x7f8b6c103090 2024-03-26T04:18:18.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:18.927+0000 7f8b69ffb700 1 -- 172.21.15.16:0/664459661 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2193381107 0 0) 0x7f8b6c1a6ca0 con 0x7f8b6c10aea0 2024-03-26T04:18:18.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:18.927+0000 7f8b69ffb700 1 -- 172.21.15.16:0/664459661 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b580031f0 con 0x7f8b6c10aea0 2024-03-26T04:18:18.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:18.927+0000 7f8b69ffb700 1 -- 172.21.15.16:0/664459661 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f8b540013b0 con 0x7f8b6c10aea0 2024-03-26T04:18:18.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:18.927+0000 7f8b69ffb700 1 -- 172.21.15.16:0/664459661 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3993238036 0 0) 0x7f8b6c1a9080 con 0x7f8b6c10e9b0 2024-03-26T04:18:18.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:18.927+0000 7f8b69ffb700 1 -- 172.21.15.16:0/664459661 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8b6c1a6ca0 con 0x7f8b6c10e9b0 2024-03-26T04:18:18.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:18.927+0000 7f8b69ffb700 1 -- 172.21.15.16:0/664459661 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f8b64001af0 con 0x7f8b6c10e9b0 2024-03-26T04:18:18.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:18.927+0000 7f8b69ffb700 1 -- 172.21.15.16:0/664459661 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1484502009 0 0) 0x7f8b6c1a7e80 con 0x7f8b6c103090 2024-03-26T04:18:18.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:18.927+0000 7f8b69ffb700 1 -- 172.21.15.16:0/664459661 >> v1:172.21.15.16:6790/0 conn(0x7f8b6c10aea0 legacy=0x7f8b6c1a1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:18.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:18.927+0000 7f8b69ffb700 1 -- 172.21.15.16:0/664459661 >> v1:172.21.15.183:6789/0 conn(0x7f8b6c10e9b0 legacy=0x7f8b6c1a5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:18.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:18.927+0000 7f8b69ffb700 1 -- 172.21.15.16:0/664459661 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b6c1aa280 con 0x7f8b6c103090 2024-03-26T04:18:18.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:18.927+0000 7f8b72d70700 1 -- 172.21.15.16:0/664459661 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b6c1a6e90 con 0x7f8b6c103090 2024-03-26T04:18:18.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:18.927+0000 7f8b69ffb700 1 -- 172.21.15.16:0/664459661 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f8b5c003b10 con 0x7f8b6c103090 2024-03-26T04:18:18.930 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:18.927+0000 7f8b69ffb700 1 -- 172.21.15.16:0/664459661 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f8b5c005bd0 con 0x7f8b6c103090 2024-03-26T04:18:18.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:18.927+0000 7f8b72d70700 1 -- 172.21.15.16:0/664459661 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b6c1a7310 con 0x7f8b6c103090 2024-03-26T04:18:18.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:18.928+0000 7f8b72d70700 1 -- 172.21.15.16:0/664459661 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b6c06a220 con 0x7f8b6c103090 2024-03-26T04:18:18.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:18.928+0000 7f8b69ffb700 1 -- 172.21.15.16:0/664459661 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 22) v1 ==== 91420+0+0 (unknown 3095259778 0 0) 0x7f8b5c003680 con 0x7f8b6c103090 2024-03-26T04:18:18.931 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:18.929+0000 7f8b69ffb700 1 -- 172.21.15.16:0/664459661 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(70..70 src has 1..70) v4 ==== 6172+0+0 (unknown 1260882646 0 0) 0x7f8b5c088900 con 0x7f8b6c103090 2024-03-26T04:18:18.934 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:18.934+0000 7f8b69ffb700 1 -- 172.21.15.16:0/664459661 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f8b5c0559d0 con 0x7f8b6c103090 2024-03-26T04:18:19.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.141+0000 7f8b72d70700 1 -- 172.21.15.16:0/664459661 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f8b6c108b60 con 0x7f8b6c103090 2024-03-26T04:18:19.142 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.141+0000 7f8b69ffb700 1 -- 172.21.15.16:0/664459661 <== mon.0 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (unknown 2131975755 0 778772972) 0x7f8b5c053fd0 con 0x7f8b6c103090 2024-03-26T04:18:19.142 INFO:teuthology.orchestra.run.smithi016.stdout:163208757274 2024-03-26T04:18:19.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.144+0000 7f8b72d70700 1 -- 172.21.15.16:0/664459661 >> v1:172.21.15.16:6800/3868004154 conn(0x7f8b5806eb50 legacy=0x7f8b58071010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:19.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.144+0000 7f8b72d70700 1 -- 172.21.15.16:0/664459661 >> v1:172.21.15.16:6789/0 conn(0x7f8b6c103090 legacy=0x7f8b6c10a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:19.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.144+0000 7f8b72d70700 1 -- 172.21.15.16:0/664459661 shutdown_connections 2024-03-26T04:18:19.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.144+0000 7f8b72d70700 1 -- 172.21.15.16:0/664459661 >> 172.21.15.16:0/664459661 conn(0x7f8b6c0fc820 msgr2=0x7f8b6c0fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:18:19.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.144+0000 7f8b72d70700 1 -- 172.21.15.16:0/664459661 shutdown_connections 2024-03-26T04:18:19.144 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.144+0000 7f8b72d70700 1 -- 172.21.15.16:0/664459661 wait complete. 2024-03-26T04:18:19.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:19 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/664459661' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T04:18:19.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:19 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/664459661' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T04:18:19.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:19 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/664459661' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-26T04:18:20.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.450+0000 7fbf458e4700 1 -- 172.21.15.16:0/2120449381 >> v1:172.21.15.16:6789/0 conn(0x7fbf40109c00 legacy=0x7fbf4010bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:20.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.450+0000 7fbf458e4700 1 -- 172.21.15.16:0/2120449381 shutdown_connections 2024-03-26T04:18:20.406 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.450+0000 7fbf458e4700 1 -- 172.21.15.16:0/2120449381 >> 172.21.15.16:0/2120449381 conn(0x7fbf400fc820 msgr2=0x7fbf400fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:18:20.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.451+0000 7fbf458e4700 1 -- 172.21.15.16:0/2120449381 shutdown_connections 2024-03-26T04:18:20.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.451+0000 7fbf458e4700 1 -- 172.21.15.16:0/2120449381 wait complete. 2024-03-26T04:18:20.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.451+0000 7fbf458e4700 1 Processor -- start 2024-03-26T04:18:20.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.451+0000 7fbf458e4700 1 -- start start 2024-03-26T04:18:20.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.451+0000 7fbf458e4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbf401a6b90 con 0x7fbf40109c00 2024-03-26T04:18:20.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.451+0000 7fbf458e4700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbf401a7d70 con 0x7fbf40100f70 2024-03-26T04:18:20.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.451+0000 7fbf458e4700 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbf401a8f70 con 0x7fbf4010d7a0 2024-03-26T04:18:20.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.452+0000 7fbf3e7fc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbf40109c00 0x7fbf401a1b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44476/0 (socket says 172.21.15.16:44476) 2024-03-26T04:18:20.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.452+0000 7fbf3f7fe700 1 --1- >> v1:172.21.15.16:6790/0 conn(0x7fbf4010d7a0 0x7fbf401a5250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6790/0 says I am v1:172.21.15.16:39616/0 (socket says 172.21.15.16:39616) 2024-03-26T04:18:20.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.452+0000 7fbf3e7fc700 1 -- 172.21.15.16:0/370893074 learned_addr learned my addr 172.21.15.16:0/370893074 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:18:20.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.452+0000 7fbf448e2700 1 -- 172.21.15.16:0/370893074 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1155676086 0 0) 0x7fbf401a6b90 con 0x7fbf40109c00 2024-03-26T04:18:20.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.452+0000 7fbf448e2700 1 -- 172.21.15.16:0/370893074 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf280035b0 con 0x7fbf40109c00 2024-03-26T04:18:20.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.452+0000 7fbf448e2700 1 -- 172.21.15.16:0/370893074 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3169627528 0 0) 0x7fbf401a8f70 con 0x7fbf4010d7a0 2024-03-26T04:18:20.407 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.452+0000 7fbf448e2700 1 -- 172.21.15.16:0/370893074 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf401a6b90 con 0x7fbf4010d7a0 2024-03-26T04:18:20.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.452+0000 7fbf448e2700 1 -- 172.21.15.16:0/370893074 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4016966162 0 0) 0x7fbf401a7d70 con 0x7fbf40100f70 2024-03-26T04:18:20.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.452+0000 7fbf448e2700 1 -- 172.21.15.16:0/370893074 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf401a8f70 con 0x7fbf40100f70 2024-03-26T04:18:20.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.452+0000 7fbf448e2700 1 -- 172.21.15.16:0/370893074 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2138648130 0 0) 0x7fbf280035b0 con 0x7fbf40109c00 2024-03-26T04:18:20.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.452+0000 7fbf448e2700 1 -- 172.21.15.16:0/370893074 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf401a7d70 con 0x7fbf40109c00 2024-03-26T04:18:20.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.452+0000 7fbf448e2700 1 -- 172.21.15.16:0/370893074 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 728291895 0 0) 0x7fbf401a6b90 con 0x7fbf4010d7a0 2024-03-26T04:18:20.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.452+0000 7fbf448e2700 1 -- 172.21.15.16:0/370893074 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf280035b0 con 0x7fbf4010d7a0 2024-03-26T04:18:20.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.452+0000 7fbf448e2700 1 -- 172.21.15.16:0/370893074 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fbf340015d0 con 0x7fbf40109c00 2024-03-26T04:18:20.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.452+0000 7fbf448e2700 1 -- 172.21.15.16:0/370893074 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fbf38001b50 con 0x7fbf4010d7a0 2024-03-26T04:18:20.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.452+0000 7fbf448e2700 1 -- 172.21.15.16:0/370893074 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 178593734 0 0) 0x7fbf401a8f70 con 0x7fbf40100f70 2024-03-26T04:18:20.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.452+0000 7fbf448e2700 1 -- 172.21.15.16:0/370893074 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf401a6b90 con 0x7fbf40100f70 2024-03-26T04:18:20.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.452+0000 7fbf448e2700 1 -- 172.21.15.16:0/370893074 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fbf300018d0 con 0x7fbf40100f70 2024-03-26T04:18:20.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.452+0000 7fbf448e2700 1 -- 172.21.15.16:0/370893074 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2011630100 0 0) 0x7fbf401a7d70 con 0x7fbf40109c00 2024-03-26T04:18:20.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.452+0000 7fbf448e2700 1 -- 172.21.15.16:0/370893074 >> v1:172.21.15.16:6790/0 conn(0x7fbf4010d7a0 legacy=0x7fbf401a5250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:20.408 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.452+0000 7fbf448e2700 1 -- 172.21.15.16:0/370893074 >> v1:172.21.15.183:6789/0 conn(0x7fbf40100f70 legacy=0x7fbf40106040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:20.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.452+0000 7fbf448e2700 1 -- 172.21.15.16:0/370893074 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf401aa170 con 0x7fbf40109c00 2024-03-26T04:18:20.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.452+0000 7fbf458e4700 1 -- 172.21.15.16:0/370893074 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf401a9160 con 0x7fbf40109c00 2024-03-26T04:18:20.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.452+0000 7fbf458e4700 1 -- 172.21.15.16:0/370893074 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf401a95e0 con 0x7fbf40109c00 2024-03-26T04:18:20.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.453+0000 7fbf448e2700 1 -- 172.21.15.16:0/370893074 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7fbf34003b10 con 0x7fbf40109c00 2024-03-26T04:18:20.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.453+0000 7fbf448e2700 1 -- 172.21.15.16:0/370893074 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fbf34004e60 con 0x7fbf40109c00 2024-03-26T04:18:20.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.453+0000 7fbf458e4700 1 -- 172.21.15.16:0/370893074 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf2c005120 con 0x7fbf40109c00 2024-03-26T04:18:20.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.454+0000 7fbf448e2700 1 -- 172.21.15.16:0/370893074 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 22) v1 ==== 91420+0+0 (unknown 3095259778 0 0) 0x7fbf34003680 con 0x7fbf40109c00 2024-03-26T04:18:20.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.454+0000 7fbf448e2700 1 -- 172.21.15.16:0/370893074 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(70..70 src has 1..70) v4 ==== 6172+0+0 (unknown 1260882646 0 0) 0x7fbf34088a60 con 0x7fbf40109c00 2024-03-26T04:18:20.409 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.459+0000 7fbf448e2700 1 -- 172.21.15.16:0/370893074 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fbf34055ea0 con 0x7fbf40109c00 2024-03-26T04:18:20.409 INFO:teuthology.orchestra.run.smithi016.stdout:231928234002 2024-03-26T04:18:20.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.684+0000 7fbf458e4700 1 -- 172.21.15.16:0/370893074 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fbf2c0053d0 con 0x7fbf40109c00 2024-03-26T04:18:20.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.685+0000 7fbf448e2700 1 -- 172.21.15.16:0/370893074 <== mon.0 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (unknown 1482059429 0 487089970) 0x7fbf34054210 con 0x7fbf40109c00 2024-03-26T04:18:20.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.687+0000 7fbf458e4700 1 -- 172.21.15.16:0/370893074 >> v1:172.21.15.16:6800/3868004154 conn(0x7fbf2806ea80 legacy=0x7fbf28070f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:20.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.687+0000 7fbf458e4700 1 -- 172.21.15.16:0/370893074 >> v1:172.21.15.16:6789/0 conn(0x7fbf40109c00 legacy=0x7fbf401a1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:20.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.687+0000 7fbf458e4700 1 -- 172.21.15.16:0/370893074 shutdown_connections 2024-03-26T04:18:20.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.687+0000 7fbf458e4700 1 -- 172.21.15.16:0/370893074 >> 172.21.15.16:0/370893074 conn(0x7fbf400fc820 msgr2=0x7fbf400ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:18:20.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.687+0000 7fbf458e4700 1 -- 172.21.15.16:0/370893074 shutdown_connections 2024-03-26T04:18:20.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:19.687+0000 7fbf458e4700 1 -- 172.21.15.16:0/370893074 wait complete. 2024-03-26T04:18:20.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:20 smithi016 ceph-mon[37215]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:18:20.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:20 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:20.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:20 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:20.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:20 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/370893074' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T04:18:20.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:20 smithi016 ceph-mon[29101]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:18:20.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:20 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:20.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:20 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:20.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:20 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/370893074' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T04:18:20.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:20 smithi183 ceph-mon[33745]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:18:20.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:20 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:20.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:20 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:20.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:20 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/370893074' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-26T04:18:20.951 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928233998 got 231928234002 for osd.7 2024-03-26T04:18:20.951 DEBUG:teuthology.parallel:result is None 2024-03-26T04:18:21.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.113+0000 7f716ee3e700 1 -- 172.21.15.16:0/4052936449 >> v1:172.21.15.16:6789/0 conn(0x7f7168102fd0 legacy=0x7f71681033b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:21.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.113+0000 7f716ee3e700 1 -- 172.21.15.16:0/4052936449 shutdown_connections 2024-03-26T04:18:21.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.113+0000 7f716ee3e700 1 -- 172.21.15.16:0/4052936449 >> 172.21.15.16:0/4052936449 conn(0x7f71680fc820 msgr2=0x7f71680fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:18:21.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.113+0000 7f716ee3e700 1 -- 172.21.15.16:0/4052936449 shutdown_connections 2024-03-26T04:18:21.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.113+0000 7f716ee3e700 1 -- 172.21.15.16:0/4052936449 wait complete. 2024-03-26T04:18:21.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.114+0000 7f716ee3e700 1 Processor -- start 2024-03-26T04:18:21.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.114+0000 7f716ee3e700 1 -- start start 2024-03-26T04:18:21.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.114+0000 7f716ee3e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f71681a6bc0 con 0x7f7168102fd0 2024-03-26T04:18:21.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.114+0000 7f716ee3e700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f71681a7da0 con 0x7f716810e8f0 2024-03-26T04:18:21.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.114+0000 7f716ee3e700 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f71681a8fa0 con 0x7f716810ade0 2024-03-26T04:18:21.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.114+0000 7f716cbda700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7168102fd0 0x7f716810a3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44494/0 (socket says 172.21.15.16:44494) 2024-03-26T04:18:21.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.114+0000 7f7167fff700 1 --1- >> v1:172.21.15.16:6790/0 conn(0x7f716810ade0 0x7f71681a1c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6790/0 says I am v1:172.21.15.16:39636/0 (socket says 172.21.15.16:39636) 2024-03-26T04:18:21.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.114+0000 7f716cbda700 1 -- 172.21.15.16:0/2824746277 learned_addr learned my addr 172.21.15.16:0/2824746277 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:18:21.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.115+0000 7f7165ffb700 1 -- 172.21.15.16:0/2824746277 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3972160942 0 0) 0x7f71681a6bc0 con 0x7f7168102fd0 2024-03-26T04:18:21.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.115+0000 7f7165ffb700 1 -- 172.21.15.16:0/2824746277 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71580035b0 con 0x7f7168102fd0 2024-03-26T04:18:21.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.115+0000 7f7165ffb700 1 -- 172.21.15.16:0/2824746277 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 485445775 0 0) 0x7f71681a8fa0 con 0x7f716810ade0 2024-03-26T04:18:21.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.115+0000 7f7165ffb700 1 -- 172.21.15.16:0/2824746277 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71681a6bc0 con 0x7f716810ade0 2024-03-26T04:18:21.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.115+0000 7f7165ffb700 1 -- 172.21.15.16:0/2824746277 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4287039392 0 0) 0x7f71681a7da0 con 0x7f716810e8f0 2024-03-26T04:18:21.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.115+0000 7f7165ffb700 1 -- 172.21.15.16:0/2824746277 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71681a8fa0 con 0x7f716810e8f0 2024-03-26T04:18:21.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.115+0000 7f7165ffb700 1 -- 172.21.15.16:0/2824746277 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 63288330 0 0) 0x7f71580035b0 con 0x7f7168102fd0 2024-03-26T04:18:21.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.115+0000 7f7165ffb700 1 -- 172.21.15.16:0/2824746277 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71681a7da0 con 0x7f7168102fd0 2024-03-26T04:18:21.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.115+0000 7f7165ffb700 1 -- 172.21.15.16:0/2824746277 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f715c004550 con 0x7f7168102fd0 2024-03-26T04:18:21.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.115+0000 7f7165ffb700 1 -- 172.21.15.16:0/2824746277 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 299627563 0 0) 0x7f71681a6bc0 con 0x7f716810ade0 2024-03-26T04:18:21.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.115+0000 7f7165ffb700 1 -- 172.21.15.16:0/2824746277 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71580035b0 con 0x7f716810ade0 2024-03-26T04:18:21.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.115+0000 7f7165ffb700 1 -- 172.21.15.16:0/2824746277 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f7154001830 con 0x7f716810ade0 2024-03-26T04:18:21.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.115+0000 7f7165ffb700 1 -- 172.21.15.16:0/2824746277 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1121256195 0 0) 0x7f71681a8fa0 con 0x7f716810e8f0 2024-03-26T04:18:21.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.115+0000 7f7165ffb700 1 -- 172.21.15.16:0/2824746277 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71681a6bc0 con 0x7f716810e8f0 2024-03-26T04:18:21.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.115+0000 7f7165ffb700 1 -- 172.21.15.16:0/2824746277 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f7160001c30 con 0x7f716810e8f0 2024-03-26T04:18:21.117 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.115+0000 7f7165ffb700 1 -- 172.21.15.16:0/2824746277 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3766202843 0 0) 0x7f71681a7da0 con 0x7f7168102fd0 2024-03-26T04:18:21.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.115+0000 7f7165ffb700 1 -- 172.21.15.16:0/2824746277 >> v1:172.21.15.16:6790/0 conn(0x7f716810ade0 legacy=0x7f71681a1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:21.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.115+0000 7f7165ffb700 1 -- 172.21.15.16:0/2824746277 >> v1:172.21.15.183:6789/0 conn(0x7f716810e8f0 legacy=0x7f71681a5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:21.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.115+0000 7f7165ffb700 1 -- 172.21.15.16:0/2824746277 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71681aa1a0 con 0x7f7168102fd0 2024-03-26T04:18:21.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.115+0000 7f716ee3e700 1 -- 172.21.15.16:0/2824746277 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71681a9190 con 0x7f7168102fd0 2024-03-26T04:18:21.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.115+0000 7f7165ffb700 1 -- 172.21.15.16:0/2824746277 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f715c001aa0 con 0x7f7168102fd0 2024-03-26T04:18:21.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.115+0000 7f7165ffb700 1 -- 172.21.15.16:0/2824746277 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f715c005040 con 0x7f7168102fd0 2024-03-26T04:18:21.118 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.115+0000 7f716ee3e700 1 -- 172.21.15.16:0/2824746277 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f71681a9610 con 0x7f7168102fd0 2024-03-26T04:18:21.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.116+0000 7f716ee3e700 1 -- 172.21.15.16:0/2824746277 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f716806a2e0 con 0x7f7168102fd0 2024-03-26T04:18:21.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.117+0000 7f7165ffb700 1 -- 172.21.15.16:0/2824746277 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 22) v1 ==== 91420+0+0 (unknown 3095259778 0 0) 0x7f715c003e70 con 0x7f7168102fd0 2024-03-26T04:18:21.119 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.117+0000 7f7165ffb700 1 -- 172.21.15.16:0/2824746277 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(70..70 src has 1..70) v4 ==== 6172+0+0 (unknown 1260882646 0 0) 0x7f715c088c30 con 0x7f7168102fd0 2024-03-26T04:18:21.122 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.122+0000 7f7165ffb700 1 -- 172.21.15.16:0/2824746277 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f715c055f90 con 0x7f7168102fd0 2024-03-26T04:18:21.324 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.323+0000 7efcba16e700 1 -- 172.21.15.16:0/2156037005 >> v1:172.21.15.16:6790/0 conn(0x7efcb410ad20 legacy=0x7efcb410d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:21.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.324+0000 7efcba16e700 1 -- 172.21.15.16:0/2156037005 shutdown_connections 2024-03-26T04:18:21.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.324+0000 7efcba16e700 1 -- 172.21.15.16:0/2156037005 >> 172.21.15.16:0/2156037005 conn(0x7efcb40fd920 msgr2=0x7efcb40ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:18:21.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.324+0000 7efcba16e700 1 -- 172.21.15.16:0/2156037005 shutdown_connections 2024-03-26T04:18:21.325 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.324+0000 7efcba16e700 1 -- 172.21.15.16:0/2156037005 wait complete. 2024-03-26T04:18:21.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.324+0000 7efcba16e700 1 Processor -- start 2024-03-26T04:18:21.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.324+0000 7efcba16e700 1 -- start start 2024-03-26T04:18:21.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.325+0000 7efcba16e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7efcb41a8f40 con 0x7efcb410e8c0 2024-03-26T04:18:21.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.325+0000 7efcba16e700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7efcb41aa120 con 0x7efcb4102090 2024-03-26T04:18:21.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.325+0000 7efcba16e700 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7efcb41ab320 con 0x7efcb410ad20 2024-03-26T04:18:21.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.325+0000 7efcb3fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7efcb410e8c0 0x7efcb41a7750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44512/0 (socket says 172.21.15.16:44512) 2024-03-26T04:18:21.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.325+0000 7efcb2ffd700 1 --1- >> v1:172.21.15.16:6790/0 conn(0x7efcb410ad20 0x7efcb41a4020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6790/0 says I am v1:172.21.15.16:39652/0 (socket says 172.21.15.16:39652) 2024-03-26T04:18:21.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.325+0000 7efcb3fff700 1 -- 172.21.15.16:0/111277602 learned_addr learned my addr 172.21.15.16:0/111277602 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:18:21.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.325+0000 7efcb0ff9700 1 -- 172.21.15.16:0/111277602 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1250043995 0 0) 0x7efcb41ab320 con 0x7efcb410ad20 2024-03-26T04:18:21.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.325+0000 7efcb0ff9700 1 -- 172.21.15.16:0/111277602 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc9c0035b0 con 0x7efcb410ad20 2024-03-26T04:18:21.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.325+0000 7efcb0ff9700 1 -- 172.21.15.16:0/111277602 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4197773612 0 0) 0x7efcb41a8f40 con 0x7efcb410e8c0 2024-03-26T04:18:21.327 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.325+0000 7efcb0ff9700 1 -- 172.21.15.16:0/111277602 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcb41ab320 con 0x7efcb410e8c0 2024-03-26T04:18:21.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.325+0000 7efcb0ff9700 1 -- 172.21.15.16:0/111277602 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3373474202 0 0) 0x7efcb41aa120 con 0x7efcb4102090 2024-03-26T04:18:21.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.325+0000 7efcb0ff9700 1 -- 172.21.15.16:0/111277602 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcb41a8f40 con 0x7efcb4102090 2024-03-26T04:18:21.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.325+0000 7efcb0ff9700 1 -- 172.21.15.16:0/111277602 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 786538126 0 0) 0x7efc9c0035b0 con 0x7efcb410ad20 2024-03-26T04:18:21.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.325+0000 7efcb0ff9700 1 -- 172.21.15.16:0/111277602 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efcb41aa120 con 0x7efcb410ad20 2024-03-26T04:18:21.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.325+0000 7efcb0ff9700 1 -- 172.21.15.16:0/111277602 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7efca80044f0 con 0x7efcb410ad20 2024-03-26T04:18:21.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.325+0000 7efcb0ff9700 1 -- 172.21.15.16:0/111277602 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3871004967 0 0) 0x7efcb41ab320 con 0x7efcb410e8c0 2024-03-26T04:18:21.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.325+0000 7efcb0ff9700 1 -- 172.21.15.16:0/111277602 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc9c0035b0 con 0x7efcb410e8c0 2024-03-26T04:18:21.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.325+0000 7efcb0ff9700 1 -- 172.21.15.16:0/111277602 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7efcac001bb0 con 0x7efcb410e8c0 2024-03-26T04:18:21.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.325+0000 7efcb0ff9700 1 -- 172.21.15.16:0/111277602 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2998838626 0 0) 0x7efcb41a8f40 con 0x7efcb4102090 2024-03-26T04:18:21.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.326+0000 7efcb0ff9700 1 -- 172.21.15.16:0/111277602 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efcb41ab320 con 0x7efcb4102090 2024-03-26T04:18:21.328 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.326+0000 7efcb0ff9700 1 -- 172.21.15.16:0/111277602 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7efca40019f0 con 0x7efcb4102090 2024-03-26T04:18:21.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.326+0000 7efcb0ff9700 1 -- 172.21.15.16:0/111277602 <== mon.2 v1:172.21.15.16:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1833043135 0 0) 0x7efcb41aa120 con 0x7efcb410ad20 2024-03-26T04:18:21.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.326+0000 7efcb0ff9700 1 -- 172.21.15.16:0/111277602 >> v1:172.21.15.183:6789/0 conn(0x7efcb4102090 legacy=0x7efcb4107270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:21.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.326+0000 7efcb0ff9700 1 -- 172.21.15.16:0/111277602 >> v1:172.21.15.16:6789/0 conn(0x7efcb410e8c0 legacy=0x7efcb41a7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:21.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.326+0000 7efcb0ff9700 1 -- 172.21.15.16:0/111277602 --> v1:172.21.15.16:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb41ac520 con 0x7efcb410ad20 2024-03-26T04:18:21.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.326+0000 7efcba16e700 1 -- 172.21.15.16:0/111277602 --> v1:172.21.15.16:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcb41a9130 con 0x7efcb410ad20 2024-03-26T04:18:21.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.326+0000 7efcba16e700 1 -- 172.21.15.16:0/111277602 --> v1:172.21.15.16:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efcb41a95b0 con 0x7efcb410ad20 2024-03-26T04:18:21.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.326+0000 7efcb0ff9700 1 -- 172.21.15.16:0/111277602 <== mon.2 v1:172.21.15.16:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7efca8001a20 con 0x7efcb410ad20 2024-03-26T04:18:21.329 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.326+0000 7efcb0ff9700 1 -- 172.21.15.16:0/111277602 <== mon.2 v1:172.21.15.16:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7efca8004da0 con 0x7efcb410ad20 2024-03-26T04:18:21.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.327+0000 7efcba16e700 1 -- 172.21.15.16:0/111277602 --> v1:172.21.15.16:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efca0005120 con 0x7efcb410ad20 2024-03-26T04:18:21.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.327+0000 7efcb0ff9700 1 -- 172.21.15.16:0/111277602 <== mon.2 v1:172.21.15.16:6790/0 7 ==== mgrmap(e 22) v1 ==== 91420+0+0 (unknown 3095259778 0 0) 0x7efca8004f00 con 0x7efcb410ad20 2024-03-26T04:18:21.330 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.328+0000 7efcb0ff9700 1 -- 172.21.15.16:0/111277602 <== mon.2 v1:172.21.15.16:6790/0 8 ==== osd_map(70..70 src has 1..70) v4 ==== 6172+0+0 (unknown 1260882646 0 0) 0x7efca8088b10 con 0x7efcb410ad20 2024-03-26T04:18:21.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.337+0000 7efcb0ff9700 1 -- 172.21.15.16:0/111277602 <== mon.2 v1:172.21.15.16:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7efca8056c70 con 0x7efcb410ad20 2024-03-26T04:18:21.337 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.337+0000 7f716ee3e700 1 -- 172.21.15.16:0/2824746277 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f7168068970 con 0x7f7168102fd0 2024-03-26T04:18:21.338 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.337+0000 7f7165ffb700 1 -- 172.21.15.16:0/2824746277 <== mon.0 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (unknown 832126871 0 1580136542) 0x7f715c054300 con 0x7f7168102fd0 2024-03-26T04:18:21.338 INFO:teuthology.orchestra.run.smithi016.stdout:55834574896 2024-03-26T04:18:21.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.340+0000 7f716ee3e700 1 -- 172.21.15.16:0/2824746277 >> v1:172.21.15.16:6800/3868004154 conn(0x7f715806eab0 legacy=0x7f7158070f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:21.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.340+0000 7f716ee3e700 1 -- 172.21.15.16:0/2824746277 >> v1:172.21.15.16:6789/0 conn(0x7f7168102fd0 legacy=0x7f716810a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:21.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.340+0000 7f716ee3e700 1 -- 172.21.15.16:0/2824746277 shutdown_connections 2024-03-26T04:18:21.340 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.340+0000 7f716ee3e700 1 -- 172.21.15.16:0/2824746277 >> 172.21.15.16:0/2824746277 conn(0x7f71680fc820 msgr2=0x7f71680fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:18:21.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.340+0000 7f716ee3e700 1 -- 172.21.15.16:0/2824746277 shutdown_connections 2024-03-26T04:18:21.341 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.340+0000 7f716ee3e700 1 -- 172.21.15.16:0/2824746277 wait complete. 2024-03-26T04:18:21.573 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:21 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:18:21.573 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:21 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:18:21.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.572+0000 7efcba16e700 1 -- 172.21.15.16:0/111277602 --> v1:172.21.15.16:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7efca00053d0 con 0x7efcb410ad20 2024-03-26T04:18:21.573 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.572+0000 7efcb0ff9700 1 -- 172.21.15.16:0/111277602 <== mon.2 v1:172.21.15.16:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (unknown 1823589948 0 1469866018) 0x7efca8054fe0 con 0x7efcb410ad20 2024-03-26T04:18:21.574 INFO:teuthology.orchestra.run.smithi016.stdout:128849018912 2024-03-26T04:18:21.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.575+0000 7efcba16e700 1 -- 172.21.15.16:0/111277602 >> v1:172.21.15.16:6800/3868004154 conn(0x7efc9c06ebb0 legacy=0x7efc9c071070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:21.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.575+0000 7efcba16e700 1 -- 172.21.15.16:0/111277602 >> v1:172.21.15.16:6790/0 conn(0x7efcb410ad20 legacy=0x7efcb41a4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:21.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.575+0000 7efcba16e700 1 -- 172.21.15.16:0/111277602 shutdown_connections 2024-03-26T04:18:21.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.575+0000 7efcba16e700 1 -- 172.21.15.16:0/111277602 >> 172.21.15.16:0/111277602 conn(0x7efcb40fd920 msgr2=0x7efcb4111a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:18:21.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.575+0000 7efcba16e700 1 -- 172.21.15.16:0/111277602 shutdown_connections 2024-03-26T04:18:21.575 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:21.575+0000 7efcba16e700 1 -- 172.21.15.16:0/111277602 wait complete. 2024-03-26T04:18:21.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:21 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:18:22.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:18:21 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:18:22.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:22 smithi016 ceph-mon[29101]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:18:22.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:22 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2824746277' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T04:18:22.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:22 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:22.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:22 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:22.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:22 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:18:22.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:22 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:18:22.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:22 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:18:22.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:22 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:18:22.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:22 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:18:22.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:22 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:18:22.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:22 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:18:22.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:22 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:18:22.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:22 smithi016 ceph-mon[29101]: Adjusting osd_memory_target on smithi016 to 3761M 2024-03-26T04:18:22.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:22 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:22.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:22 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:18:22.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:22 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:18:22.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:22 smithi016 ceph-mon[29101]: Updating smithi016:/etc/ceph/ceph.conf 2024-03-26T04:18:22.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:22 smithi016 ceph-mon[29101]: Updating smithi183:/etc/ceph/ceph.conf 2024-03-26T04:18:22.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:22 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/111277602' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T04:18:22.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:22 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:22.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:22 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:22.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:22 smithi016 ceph-mon[37215]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:18:22.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:22 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2824746277' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T04:18:22.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:22 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:22.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:22 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:22.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:22 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:18:22.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:22 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:18:22.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:22 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:18:22.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:22 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:18:22.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:22 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:18:22.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:22 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:18:22.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:22 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:18:22.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:22 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:18:22.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:22 smithi016 ceph-mon[37215]: Adjusting osd_memory_target on smithi016 to 3761M 2024-03-26T04:18:22.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:22 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:22.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:22 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:18:22.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:22 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:18:22.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:22 smithi016 ceph-mon[37215]: Updating smithi016:/etc/ceph/ceph.conf 2024-03-26T04:18:22.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:22 smithi016 ceph-mon[37215]: Updating smithi183:/etc/ceph/ceph.conf 2024-03-26T04:18:22.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:22 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/111277602' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T04:18:22.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:22 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:22.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:22 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:22.735 INFO:tasks.cephadm.ceph_manager.ceph:need seq 128849018909 got 128849018912 for osd.4 2024-03-26T04:18:22.735 DEBUG:teuthology.parallel:result is None 2024-03-26T04:18:22.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:22 smithi183 ceph-mon[33745]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:18:22.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:22 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2824746277' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-26T04:18:22.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:22 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:22.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:22 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:22.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:22 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:18:22.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:22 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:18:22.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:22 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:18:22.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:22 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:18:22.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:22 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:18:22.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:22 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:18:22.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:22 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:18:22.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:22 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:18:22.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:22 smithi183 ceph-mon[33745]: Adjusting osd_memory_target on smithi016 to 3761M 2024-03-26T04:18:22.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:22 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:22.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:22 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:18:22.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:22 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:18:22.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:22 smithi183 ceph-mon[33745]: Updating smithi016:/etc/ceph/ceph.conf 2024-03-26T04:18:22.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:22 smithi183 ceph-mon[33745]: Updating smithi183:/etc/ceph/ceph.conf 2024-03-26T04:18:22.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:22 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/111277602' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-26T04:18:22.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:22 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:22.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:22 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:23.091 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574892 got 55834574896 for osd.1 2024-03-26T04:18:23.091 DEBUG:teuthology.parallel:result is None 2024-03-26T04:18:23.272 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:18:23.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.121+0000 7f79c1dca700 1 -- 172.21.15.16:0/1366133848 >> v1:172.21.15.16:6790/0 conn(0x7f79bc10acb0 legacy=0x7f79bc10d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:23.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.121+0000 7f79c1dca700 1 -- 172.21.15.16:0/1366133848 shutdown_connections 2024-03-26T04:18:23.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.121+0000 7f79c1dca700 1 -- 172.21.15.16:0/1366133848 >> 172.21.15.16:0/1366133848 conn(0x7f79bc0771b0 msgr2=0x7f79bc0fe530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:18:23.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.121+0000 7f79c1dca700 1 -- 172.21.15.16:0/1366133848 shutdown_connections 2024-03-26T04:18:23.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.121+0000 7f79c1dca700 1 -- 172.21.15.16:0/1366133848 wait complete. 2024-03-26T04:18:23.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.122+0000 7f79c1dca700 1 Processor -- start 2024-03-26T04:18:23.476 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.122+0000 7f79c1dca700 1 -- start start 2024-03-26T04:18:23.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.122+0000 7f79c1dca700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f79bc19eb30 con 0x7f79bc106e10 2024-03-26T04:18:23.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.123+0000 7f79c1dca700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f79bc19ecc0 con 0x7f79bc10acb0 2024-03-26T04:18:23.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.123+0000 7f79bbfff700 1 --1- >> v1:172.21.15.16:6790/0 conn(0x7f79bc10e9b0 0x7f79bc19e420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6790/0 says I am v1:172.21.15.16:39676/0 (socket says 172.21.15.16:39676) 2024-03-26T04:18:23.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.123+0000 7f79bbfff700 1 -- 172.21.15.16:0/1971721783 learned_addr learned my addr 172.21.15.16:0/1971721783 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:18:23.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.123+0000 7f79b2ffd700 1 --1- 172.21.15.16:0/1971721783 >> v1:172.21.15.183:6789/0 conn(0x7f79bc10acb0 0x7f79bc19dd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.16:45368/0 (socket says 172.21.15.16:45368) 2024-03-26T04:18:23.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.123+0000 7f79c1dca700 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f79bc19ee90 con 0x7f79bc10e9b0 2024-03-26T04:18:23.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.123+0000 7f79b97fa700 1 -- 172.21.15.16:0/1971721783 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2342078960 0 0) 0x7f79bc19ecc0 con 0x7f79bc10acb0 2024-03-26T04:18:23.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.123+0000 7f79b97fa700 1 -- 172.21.15.16:0/1971721783 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79a80035b0 con 0x7f79bc10acb0 2024-03-26T04:18:23.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.123+0000 7f79b97fa700 1 -- 172.21.15.16:0/1971721783 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 290459528 0 0) 0x7f79bc19eb30 con 0x7f79bc106e10 2024-03-26T04:18:23.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.123+0000 7f79b97fa700 1 -- 172.21.15.16:0/1971721783 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79bc19ecc0 con 0x7f79bc106e10 2024-03-26T04:18:23.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.123+0000 7f79b97fa700 1 -- 172.21.15.16:0/1971721783 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3375341924 0 0) 0x7f79bc19ee90 con 0x7f79bc10e9b0 2024-03-26T04:18:23.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.123+0000 7f79b97fa700 1 -- 172.21.15.16:0/1971721783 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79bc19eb30 con 0x7f79bc10e9b0 2024-03-26T04:18:23.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.123+0000 7f79b97fa700 1 -- 172.21.15.16:0/1971721783 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3136701555 0 0) 0x7f79a80035b0 con 0x7f79bc10acb0 2024-03-26T04:18:23.477 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.123+0000 7f79b97fa700 1 -- 172.21.15.16:0/1971721783 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79bc19ee90 con 0x7f79bc10acb0 2024-03-26T04:18:23.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.123+0000 7f79b97fa700 1 -- 172.21.15.16:0/1971721783 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f79a4001510 con 0x7f79bc10acb0 2024-03-26T04:18:23.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.123+0000 7f79b97fa700 1 -- 172.21.15.16:0/1971721783 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 978941111 0 0) 0x7f79bc19ecc0 con 0x7f79bc106e10 2024-03-26T04:18:23.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.123+0000 7f79b97fa700 1 -- 172.21.15.16:0/1971721783 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79a80035b0 con 0x7f79bc106e10 2024-03-26T04:18:23.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.123+0000 7f79b97fa700 1 -- 172.21.15.16:0/1971721783 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f79ac001870 con 0x7f79bc106e10 2024-03-26T04:18:23.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.123+0000 7f79b97fa700 1 -- 172.21.15.16:0/1971721783 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3353420587 0 0) 0x7f79bc19eb30 con 0x7f79bc10e9b0 2024-03-26T04:18:23.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.123+0000 7f79b97fa700 1 -- 172.21.15.16:0/1971721783 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79bc19ecc0 con 0x7f79bc10e9b0 2024-03-26T04:18:23.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.123+0000 7f79b97fa700 1 -- 172.21.15.16:0/1971721783 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f79b4001cd0 con 0x7f79bc10e9b0 2024-03-26T04:18:23.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.123+0000 7f79b97fa700 1 -- 172.21.15.16:0/1971721783 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1070236534 0 0) 0x7f79bc19ee90 con 0x7f79bc10acb0 2024-03-26T04:18:23.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.123+0000 7f79b97fa700 1 -- 172.21.15.16:0/1971721783 >> v1:172.21.15.16:6790/0 conn(0x7f79bc10e9b0 legacy=0x7f79bc19e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:23.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.123+0000 7f79b97fa700 1 -- 172.21.15.16:0/1971721783 >> v1:172.21.15.16:6789/0 conn(0x7f79bc106e10 legacy=0x7f79bc1061b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:23.478 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.123+0000 7f79b97fa700 1 -- 172.21.15.16:0/1971721783 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79bc1ad950 con 0x7f79bc10acb0 2024-03-26T04:18:23.479 INFO:teuthology.orchestra.run.smithi016.stdout:197568495639 2024-03-26T04:18:23.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.124+0000 7f79c1dca700 1 -- 172.21.15.16:0/1971721783 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79bc1adba0 con 0x7f79bc10acb0 2024-03-26T04:18:23.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.124+0000 7f79c1dca700 1 -- 172.21.15.16:0/1971721783 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f79bc1ae020 con 0x7f79bc10acb0 2024-03-26T04:18:23.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.124+0000 7f79b97fa700 1 -- 172.21.15.16:0/1971721783 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f79a4001ab0 con 0x7f79bc10acb0 2024-03-26T04:18:23.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.124+0000 7f79b97fa700 1 -- 172.21.15.16:0/1971721783 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f79a4004f30 con 0x7f79bc10acb0 2024-03-26T04:18:23.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.125+0000 7f79c1dca700 1 -- 172.21.15.16:0/1971721783 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f799c005120 con 0x7f79bc10acb0 2024-03-26T04:18:23.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.125+0000 7f79b97fa700 1 -- 172.21.15.16:0/1971721783 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 22) v1 ==== 91420+0+0 (unknown 3095259778 0 0) 0x7f79a40037f0 con 0x7f79bc10acb0 2024-03-26T04:18:23.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.126+0000 7f79b97fa700 1 -- 172.21.15.16:0/1971721783 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(70..70 src has 1..70) v4 ==== 6172+0+0 (unknown 1260882646 0 0) 0x7f79a4088a40 con 0x7f79bc10acb0 2024-03-26T04:18:23.479 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.130+0000 7f79b97fa700 1 -- 172.21.15.16:0/1971721783 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f79a4055d40 con 0x7f79bc10acb0 2024-03-26T04:18:23.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.345+0000 7f79c1dca700 1 -- 172.21.15.16:0/1971721783 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f799c0053d0 con 0x7f79bc10acb0 2024-03-26T04:18:23.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.349+0000 7f79b97fa700 1 -- 172.21.15.16:0/1971721783 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (unknown 1274345170 0 393369010) 0x7f79a40540b0 con 0x7f79bc10acb0 2024-03-26T04:18:23.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.352+0000 7f79c1dca700 1 -- 172.21.15.16:0/1971721783 >> v1:172.21.15.16:6800/3868004154 conn(0x7f79a806eb30 legacy=0x7f79a8070ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:23.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.352+0000 7f79c1dca700 1 -- 172.21.15.16:0/1971721783 >> v1:172.21.15.183:6789/0 conn(0x7f79bc10acb0 legacy=0x7f79bc19dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:23.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.352+0000 7f79c1dca700 1 -- 172.21.15.16:0/1971721783 shutdown_connections 2024-03-26T04:18:23.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.352+0000 7f79c1dca700 1 -- 172.21.15.16:0/1971721783 >> 172.21.15.16:0/1971721783 conn(0x7f79bc0771b0 msgr2=0x7f79bc111620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:18:23.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.352+0000 7f79c1dca700 1 -- 172.21.15.16:0/1971721783 shutdown_connections 2024-03-26T04:18:23.480 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:22.352+0000 7f79c1dca700 1 -- 172.21.15.16:0/1971721783 wait complete. 2024-03-26T04:18:23.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:23 smithi016 ceph-mon[37215]: Updating smithi016:/var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:18:23.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:23 smithi016 ceph-mon[37215]: Updating smithi183:/var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:18:23.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:23 smithi016 ceph-mon[37215]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:18:23.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:23 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:18:23.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:23 smithi016 ceph-mon[37215]: Updating smithi183:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:18:23.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:23 smithi016 ceph-mon[37215]: Updating smithi016:/var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:18:23.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:23 smithi016 ceph-mon[37215]: Updating smithi183:/var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:18:23.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:23 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1971721783' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T04:18:23.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:23 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:23.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:23 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:23.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:23 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:23.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:23 smithi016 ceph-mon[37215]: Deploying daemon alertmanager.a on smithi016 2024-03-26T04:18:23.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:23 smithi016 ceph-mon[29101]: Updating smithi016:/var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:18:23.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:23 smithi016 ceph-mon[29101]: Updating smithi183:/var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:18:23.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:23 smithi016 ceph-mon[29101]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:18:23.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:23 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:18:23.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:23 smithi016 ceph-mon[29101]: Updating smithi183:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:18:23.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:23 smithi016 ceph-mon[29101]: Updating smithi016:/var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:18:23.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:23 smithi016 ceph-mon[29101]: Updating smithi183:/var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:18:23.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:23 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1971721783' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T04:18:23.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:23 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:23.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:23 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:23.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:23 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:23.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:23 smithi016 ceph-mon[29101]: Deploying daemon alertmanager.a on smithi016 2024-03-26T04:18:23.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:23 smithi183 ceph-mon[33745]: Updating smithi016:/var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:18:23.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:23 smithi183 ceph-mon[33745]: Updating smithi183:/var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/config/ceph.conf 2024-03-26T04:18:23.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:23 smithi183 ceph-mon[33745]: Updating smithi016:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:18:23.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:23 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:18:23.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:23 smithi183 ceph-mon[33745]: Updating smithi183:/etc/ceph/ceph.client.admin.keyring 2024-03-26T04:18:23.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:23 smithi183 ceph-mon[33745]: Updating smithi016:/var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:18:23.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:23 smithi183 ceph-mon[33745]: Updating smithi183:/var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/config/ceph.client.admin.keyring 2024-03-26T04:18:23.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:23 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1971721783' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-26T04:18:23.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:23 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:23.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:23 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:23.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:23 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:23.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:23 smithi183 ceph-mon[33745]: Deploying daemon alertmanager.a on smithi016 2024-03-26T04:18:24.030 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495636 got 197568495639 for osd.6 2024-03-26T04:18:24.030 DEBUG:teuthology.parallel:result is None 2024-03-26T04:18:24.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:24 smithi016 ceph-mon[37215]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:18:24.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:24 smithi016 ceph-mon[29101]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:18:24.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:24 smithi183 ceph-mon[33745]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:18:25.077 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757271 got 163208757274 for osd.5 2024-03-26T04:18:25.077 DEBUG:teuthology.parallel:result is None 2024-03-26T04:18:26.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.023+0000 7f3d21bd5700 1 -- 172.21.15.16:0/1415937775 >> v1:172.21.15.16:6789/0 conn(0x7f3d1c10e950 legacy=0x7f3d1c110df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:26.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.023+0000 7f3d21bd5700 1 -- 172.21.15.16:0/1415937775 shutdown_connections 2024-03-26T04:18:26.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.023+0000 7f3d21bd5700 1 -- 172.21.15.16:0/1415937775 >> 172.21.15.16:0/1415937775 conn(0x7f3d1c076070 msgr2=0x7f3d1c076480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:18:26.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.023+0000 7f3d21bd5700 1 -- 172.21.15.16:0/1415937775 shutdown_connections 2024-03-26T04:18:26.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.023+0000 7f3d21bd5700 1 -- 172.21.15.16:0/1415937775 wait complete. 2024-03-26T04:18:26.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.023+0000 7f3d21bd5700 1 Processor -- start 2024-03-26T04:18:26.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.023+0000 7f3d21bd5700 1 -- start start 2024-03-26T04:18:26.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.024+0000 7f3d21bd5700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3d1c1a6cd0 con 0x7f3d1c10e950 2024-03-26T04:18:26.024 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.024+0000 7f3d21bd5700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3d1c1a7eb0 con 0x7f3d1c100850 2024-03-26T04:18:26.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.024+0000 7f3d21bd5700 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f3d1c1a90b0 con 0x7f3d1c10ade0 2024-03-26T04:18:26.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.024+0000 7f3d1bfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3d1c10e950 0x7f3d1c1a5410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44982/0 (socket says 172.21.15.16:44982) 2024-03-26T04:18:26.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.024+0000 7f3d1affd700 1 --1- >> v1:172.21.15.16:6790/0 conn(0x7f3d1c10ade0 0x7f3d1c1a1ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6790/0 says I am v1:172.21.15.16:38844/0 (socket says 172.21.15.16:38844) 2024-03-26T04:18:26.025 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.024+0000 7f3d1bfff700 1 -- 172.21.15.16:0/757876265 learned_addr learned my addr 172.21.15.16:0/757876265 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:18:26.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.024+0000 7f3d18ff9700 1 -- 172.21.15.16:0/757876265 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4246295456 0 0) 0x7f3d1c1a6cd0 con 0x7f3d1c10e950 2024-03-26T04:18:26.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.024+0000 7f3d18ff9700 1 -- 172.21.15.16:0/757876265 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d040031f0 con 0x7f3d1c10e950 2024-03-26T04:18:26.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.024+0000 7f3d18ff9700 1 -- 172.21.15.16:0/757876265 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3448599026 0 0) 0x7f3d1c1a90b0 con 0x7f3d1c10ade0 2024-03-26T04:18:26.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.024+0000 7f3d18ff9700 1 -- 172.21.15.16:0/757876265 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d1c1a6cd0 con 0x7f3d1c10ade0 2024-03-26T04:18:26.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.024+0000 7f3d18ff9700 1 -- 172.21.15.16:0/757876265 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1043357786 0 0) 0x7f3d1c1a7eb0 con 0x7f3d1c100850 2024-03-26T04:18:26.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.024+0000 7f3d18ff9700 1 -- 172.21.15.16:0/757876265 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d1c1a90b0 con 0x7f3d1c100850 2024-03-26T04:18:26.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.024+0000 7f3d18ff9700 1 -- 172.21.15.16:0/757876265 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3711210428 0 0) 0x7f3d040031f0 con 0x7f3d1c10e950 2024-03-26T04:18:26.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.024+0000 7f3d18ff9700 1 -- 172.21.15.16:0/757876265 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d1c1a7eb0 con 0x7f3d1c10e950 2024-03-26T04:18:26.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.024+0000 7f3d18ff9700 1 -- 172.21.15.16:0/757876265 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2183370316 0 0) 0x7f3d1c1a6cd0 con 0x7f3d1c10ade0 2024-03-26T04:18:26.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.024+0000 7f3d18ff9700 1 -- 172.21.15.16:0/757876265 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d040031f0 con 0x7f3d1c10ade0 2024-03-26T04:18:26.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.024+0000 7f3d18ff9700 1 -- 172.21.15.16:0/757876265 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f3d14001950 con 0x7f3d1c10e950 2024-03-26T04:18:26.027 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.024+0000 7f3d18ff9700 1 -- 172.21.15.16:0/757876265 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f3d100017b0 con 0x7f3d1c10ade0 2024-03-26T04:18:26.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.024+0000 7f3d18ff9700 1 -- 172.21.15.16:0/757876265 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4278574729 0 0) 0x7f3d1c1a90b0 con 0x7f3d1c100850 2024-03-26T04:18:26.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.024+0000 7f3d18ff9700 1 -- 172.21.15.16:0/757876265 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d1c1a6cd0 con 0x7f3d1c100850 2024-03-26T04:18:26.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.024+0000 7f3d18ff9700 1 -- 172.21.15.16:0/757876265 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f3d0c0018d0 con 0x7f3d1c100850 2024-03-26T04:18:26.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.025+0000 7f3d18ff9700 1 -- 172.21.15.16:0/757876265 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 509893768 0 0) 0x7f3d1c1a7eb0 con 0x7f3d1c10e950 2024-03-26T04:18:26.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.025+0000 7f3d18ff9700 1 -- 172.21.15.16:0/757876265 >> v1:172.21.15.16:6790/0 conn(0x7f3d1c10ade0 legacy=0x7f3d1c1a1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:26.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.025+0000 7f3d18ff9700 1 -- 172.21.15.16:0/757876265 >> v1:172.21.15.183:6789/0 conn(0x7f3d1c100850 legacy=0x7f3d1c107100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:26.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.025+0000 7f3d18ff9700 1 -- 172.21.15.16:0/757876265 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d1c1aa2b0 con 0x7f3d1c10e950 2024-03-26T04:18:26.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.025+0000 7f3d21bd5700 1 -- 172.21.15.16:0/757876265 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d1c1a92a0 con 0x7f3d1c10e950 2024-03-26T04:18:26.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.025+0000 7f3d21bd5700 1 -- 172.21.15.16:0/757876265 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d1c1a97a0 con 0x7f3d1c10e950 2024-03-26T04:18:26.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.025+0000 7f3d18ff9700 1 -- 172.21.15.16:0/757876265 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f3d140014b0 con 0x7f3d1c10e950 2024-03-26T04:18:26.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.025+0000 7f3d18ff9700 1 -- 172.21.15.16:0/757876265 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f3d14005230 con 0x7f3d1c10e950 2024-03-26T04:18:26.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.026+0000 7f3d21bd5700 1 -- 172.21.15.16:0/757876265 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d08005150 con 0x7f3d1c10e950 2024-03-26T04:18:26.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.026+0000 7f3d18ff9700 1 -- 172.21.15.16:0/757876265 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 22) v1 ==== 91420+0+0 (unknown 3095259778 0 0) 0x7f3d14005470 con 0x7f3d1c10e950 2024-03-26T04:18:26.028 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.027+0000 7f3d18ff9700 1 -- 172.21.15.16:0/757876265 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(70..70 src has 1..70) v4 ==== 6172+0+0 (unknown 1260882646 0 0) 0x7f3d14088e60 con 0x7f3d1c10e950 2024-03-26T04:18:26.032 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.032+0000 7f3d18ff9700 1 -- 172.21.15.16:0/757876265 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f3d140561c0 con 0x7f3d1c10e950 2024-03-26T04:18:26.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.256+0000 7f3d21bd5700 1 -- 172.21.15.16:0/757876265 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f3d08005400 con 0x7f3d1c10e950 2024-03-26T04:18:26.256 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.256+0000 7f3d18ff9700 1 -- 172.21.15.16:0/757876265 <== mon.0 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (unknown 383520633 0 3130416629) 0x7f3d14054530 con 0x7f3d1c10e950 2024-03-26T04:18:26.257 INFO:teuthology.orchestra.run.smithi016.stdout:107374182439 2024-03-26T04:18:26.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.258+0000 7f3d21bd5700 1 -- 172.21.15.16:0/757876265 >> v1:172.21.15.16:6800/3868004154 conn(0x7f3d0406ea30 legacy=0x7f3d04070ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:26.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.258+0000 7f3d21bd5700 1 -- 172.21.15.16:0/757876265 >> v1:172.21.15.16:6789/0 conn(0x7f3d1c10e950 legacy=0x7f3d1c1a5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:26.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.258+0000 7f3d21bd5700 1 -- 172.21.15.16:0/757876265 shutdown_connections 2024-03-26T04:18:26.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.258+0000 7f3d21bd5700 1 -- 172.21.15.16:0/757876265 >> 172.21.15.16:0/757876265 conn(0x7f3d1c076070 msgr2=0x7f3d1c112060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:18:26.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.258+0000 7f3d21bd5700 1 -- 172.21.15.16:0/757876265 shutdown_connections 2024-03-26T04:18:26.259 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:26.258+0000 7f3d21bd5700 1 -- 172.21.15.16:0/757876265 wait complete. 2024-03-26T04:18:26.571 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:26 smithi016 ceph-mon[29101]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:18:26.573 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:26 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/757876265' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T04:18:26.573 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:26 smithi016 ceph-mon[37215]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:18:26.573 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:26 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/757876265' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T04:18:26.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:26 smithi183 ceph-mon[33745]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:18:26.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:26 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/757876265' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-26T04:18:27.223 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182435 got 107374182439 for osd.3 2024-03-26T04:18:27.223 DEBUG:teuthology.parallel:result is None 2024-03-26T04:18:27.224 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-26T04:18:27.224 DEBUG:teuthology.orchestra.run.smithi016:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T04:18:27.555 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:18:28.725 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:28 smithi016 ceph-mon[37215]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:18:28.725 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:28 smithi016 ceph-mon[29101]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:18:28.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:28 smithi183 ceph-mon[33745]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:18:29.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:18:28 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:18:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:18:30.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:30 smithi016 ceph-mon[29101]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:18:30.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:30 smithi016 ceph-mon[37215]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:18:30.709 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:30.664+0000 7f76618f4700 1 -- 172.21.15.16:0/1672284798 >> v1:172.21.15.16:6790/0 conn(0x7f765c10e960 legacy=0x7f765c110db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:30.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:30.664+0000 7f76618f4700 1 -- 172.21.15.16:0/1672284798 shutdown_connections 2024-03-26T04:18:30.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:30.664+0000 7f76618f4700 1 -- 172.21.15.16:0/1672284798 >> 172.21.15.16:0/1672284798 conn(0x7f765c0fc810 msgr2=0x7f765c0fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:18:30.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:30.664+0000 7f76618f4700 1 -- 172.21.15.16:0/1672284798 shutdown_connections 2024-03-26T04:18:30.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:30.664+0000 7f76618f4700 1 -- 172.21.15.16:0/1672284798 wait complete. 2024-03-26T04:18:30.710 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:30.665+0000 7f76618f4700 1 Processor -- start 2024-03-26T04:18:30.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:30.665+0000 7f76618f4700 1 -- start start 2024-03-26T04:18:30.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:30.665+0000 7f76618f4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f765c1a6b20 con 0x7f765c103040 2024-03-26T04:18:30.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:30.665+0000 7f76618f4700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f765c1a7d00 con 0x7f765c10e960 2024-03-26T04:18:30.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:30.665+0000 7f76618f4700 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f765c1a8f00 con 0x7f765c10ae50 2024-03-26T04:18:30.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:30.665+0000 7f765a7fc700 1 --1- >> v1:172.21.15.16:6790/0 conn(0x7f765c10ae50 0x7f765c1a1c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6790/0 says I am v1:172.21.15.16:38872/0 (socket says 172.21.15.16:38872) 2024-03-26T04:18:30.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:30.665+0000 7f765a7fc700 1 -- 172.21.15.16:0/179176198 learned_addr learned my addr 172.21.15.16:0/179176198 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:18:30.711 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:30.666+0000 7f76608f2700 1 -- 172.21.15.16:0/179176198 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1133929397 0 0) 0x7f765c1a8f00 con 0x7f765c10ae50 2024-03-26T04:18:30.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:30.666+0000 7f76608f2700 1 -- 172.21.15.16:0/179176198 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76440037f0 con 0x7f765c10ae50 2024-03-26T04:18:30.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:30.666+0000 7f76608f2700 1 -- 172.21.15.16:0/179176198 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3471711531 0 0) 0x7f765c1a6b20 con 0x7f765c103040 2024-03-26T04:18:30.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:30.666+0000 7f76608f2700 1 -- 172.21.15.16:0/179176198 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f765c1a8f00 con 0x7f765c103040 2024-03-26T04:18:30.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:30.666+0000 7f76608f2700 1 -- 172.21.15.16:0/179176198 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 348830577 0 0) 0x7f76440037f0 con 0x7f765c10ae50 2024-03-26T04:18:30.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:30.666+0000 7f76608f2700 1 -- 172.21.15.16:0/179176198 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f765c1a6b20 con 0x7f765c10ae50 2024-03-26T04:18:30.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:30.666+0000 7f76608f2700 1 -- 172.21.15.16:0/179176198 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f76500019c0 con 0x7f765c10ae50 2024-03-26T04:18:30.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:30.666+0000 7f76608f2700 1 -- 172.21.15.16:0/179176198 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 880895556 0 0) 0x7f765c1a7d00 con 0x7f765c10e960 2024-03-26T04:18:30.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:30.666+0000 7f76608f2700 1 -- 172.21.15.16:0/179176198 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76440037f0 con 0x7f765c10e960 2024-03-26T04:18:30.712 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:30.666+0000 7f76608f2700 1 -- 172.21.15.16:0/179176198 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2551308967 0 0) 0x7f765c1a8f00 con 0x7f765c103040 2024-03-26T04:18:30.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:30.666+0000 7f76608f2700 1 -- 172.21.15.16:0/179176198 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f765c1a7d00 con 0x7f765c103040 2024-03-26T04:18:30.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:30.666+0000 7f76608f2700 1 -- 172.21.15.16:0/179176198 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f764c001830 con 0x7f765c103040 2024-03-26T04:18:30.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:30.666+0000 7f76608f2700 1 -- 172.21.15.16:0/179176198 <== mon.2 v1:172.21.15.16:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2524611822 0 0) 0x7f765c1a6b20 con 0x7f765c10ae50 2024-03-26T04:18:30.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:30.666+0000 7f76608f2700 1 -- 172.21.15.16:0/179176198 >> v1:172.21.15.183:6789/0 conn(0x7f765c10e960 legacy=0x7f765c1a5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:30.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:30.666+0000 7f76608f2700 1 -- 172.21.15.16:0/179176198 >> v1:172.21.15.16:6789/0 conn(0x7f765c103040 legacy=0x7f765c10a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:30.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:30.666+0000 7f76608f2700 1 -- 172.21.15.16:0/179176198 --> v1:172.21.15.16:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f765c1aa100 con 0x7f765c10ae50 2024-03-26T04:18:30.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:30.666+0000 7f76618f4700 1 -- 172.21.15.16:0/179176198 --> v1:172.21.15.16:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f765c1a90f0 con 0x7f765c10ae50 2024-03-26T04:18:30.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:30.666+0000 7f76618f4700 1 -- 172.21.15.16:0/179176198 --> v1:172.21.15.16:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f765c1a9570 con 0x7f765c10ae50 2024-03-26T04:18:30.713 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:30.666+0000 7f76608f2700 1 -- 172.21.15.16:0/179176198 <== mon.2 v1:172.21.15.16:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f7650003b30 con 0x7f765c10ae50 2024-03-26T04:18:30.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:30.666+0000 7f76608f2700 1 -- 172.21.15.16:0/179176198 <== mon.2 v1:172.21.15.16:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f7650005cf0 con 0x7f765c10ae50 2024-03-26T04:18:30.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:30.668+0000 7f76618f4700 1 -- 172.21.15.16:0/179176198 --> v1:172.21.15.16:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7648005120 con 0x7f765c10ae50 2024-03-26T04:18:30.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:30.668+0000 7f76608f2700 1 -- 172.21.15.16:0/179176198 <== mon.2 v1:172.21.15.16:6790/0 7 ==== mgrmap(e 22) v1 ==== 91420+0+0 (unknown 3095259778 0 0) 0x7f765001c3a0 con 0x7f765c10ae50 2024-03-26T04:18:30.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:30.668+0000 7f76608f2700 1 -- 172.21.15.16:0/179176198 <== mon.2 v1:172.21.15.16:6790/0 8 ==== osd_map(70..70 src has 1..70) v4 ==== 6172+0+0 (unknown 1260882646 0 0) 0x7f76500889a0 con 0x7f765c10ae50 2024-03-26T04:18:30.714 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:30.673+0000 7f76608f2700 1 -- 172.21.15.16:0/179176198 <== mon.2 v1:172.21.15.16:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f7650055d00 con 0x7f765c10ae50 2024-03-26T04:18:30.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:30 smithi183 ceph-mon[33745]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:18:30.903 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:30.903+0000 7f76618f4700 1 -- 172.21.15.16:0/179176198 --> v1:172.21.15.16:6800/3868004154 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7648000b90 con 0x7f764406eb60 2024-03-26T04:18:30.913 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:30.913+0000 7f76608f2700 1 -- 172.21.15.16:0/179176198 <== mgr.14616 v1:172.21.15.16:6800/3868004154 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+346507 (unknown 2965378022 0 1285959673) 0x7f7648000b90 con 0x7f764406eb60 2024-03-26T04:18:30.918 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:18:30.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:30.917+0000 7f76618f4700 1 -- 172.21.15.16:0/179176198 >> v1:172.21.15.16:6800/3868004154 conn(0x7f764406eb60 legacy=0x7f7644071020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:30.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:30.917+0000 7f76618f4700 1 -- 172.21.15.16:0/179176198 >> v1:172.21.15.16:6790/0 conn(0x7f765c10ae50 legacy=0x7f765c1a1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:30.920 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:30.918+0000 7f76618f4700 1 -- 172.21.15.16:0/179176198 shutdown_connections 2024-03-26T04:18:30.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:30.918+0000 7f76618f4700 1 -- 172.21.15.16:0/179176198 >> 172.21.15.16:0/179176198 conn(0x7f765c0fc810 msgr2=0x7f765c0fec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:18:30.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:30.918+0000 7f76618f4700 1 -- 172.21.15.16:0/179176198 shutdown_connections 2024-03-26T04:18:30.921 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:30.918+0000 7f76618f4700 1 -- 172.21.15.16:0/179176198 wait complete. 2024-03-26T04:18:30.921 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-03-26T04:18:32.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:18:31 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:18:32.740 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:32 smithi016 ceph-mon[37215]: from='client.34640 v1:172.21.15.16:0/179176198' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:18:32.740 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:32 smithi016 ceph-mon[37215]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:18:32.740 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:32 smithi016 ceph-mon[29101]: from='client.34640 v1:172.21.15.16:0/179176198' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:18:32.740 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:32 smithi016 ceph-mon[29101]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:18:32.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:32 smithi183 ceph-mon[33745]: from='client.34640 v1:172.21.15.16:0/179176198' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:18:32.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:32 smithi183 ceph-mon[33745]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:18:33.362 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-03-26T04:18:28.951494+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1291,"num_read_kb":1105,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":794,"ondisk_log_size":794,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":244148,"kb_used_data":7196,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":749717068,"statfs":{"total":767960285184,"available":767710277632,"internally_reserved":0,"allocated":7368704,"data_stored":3927114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003072"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.145703+0000","last_change":"2024-03-26T04:17:17.170196+0000","last_active":"2024-03-26T04:17:51.145703+0000","last_peered":"2024-03-26T04:17:51.145703+0000","last_clean":"2024-03-26T04:17:51.145703+0000","last_became_active":"2024-03-26T04:17:17.169900+0000","last_became_peered":"2024-03-26T04:17:17.169900+0000","last_unstale":"2024-03-26T04:17:51.145703+0000","last_undegraded":"2024-03-26T04:17:51.145703+0000","last_fullsized":"2024-03-26T04:17:51.145703+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:50:37.434494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.239613+0000","last_change":"2024-03-26T04:17:15.171595+0000","last_active":"2024-03-26T04:17:51.239613+0000","last_peered":"2024-03-26T04:17:51.239613+0000","last_clean":"2024-03-26T04:17:51.239613+0000","last_became_active":"2024-03-26T04:17:15.171277+0000","last_became_peered":"2024-03-26T04:17:15.171277+0000","last_unstale":"2024-03-26T04:17:51.239613+0000","last_undegraded":"2024-03-26T04:17:51.239613+0000","last_fullsized":"2024-03-26T04:17:51.239613+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:28:30.163211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"61'1","reported_seq":35,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.929898+0000","last_change":"2024-03-26T04:17:11.507950+0000","last_active":"2024-03-26T04:17:50.929898+0000","last_peered":"2024-03-26T04:17:50.929898+0000","last_clean":"2024-03-26T04:17:50.929898+0000","last_became_active":"2024-03-26T04:17:11.507622+0000","last_became_peered":"2024-03-26T04:17:11.507622+0000","last_unstale":"2024-03-26T04:17:50.929898+0000","last_undegraded":"2024-03-26T04:17:50.929898+0000","last_fullsized":"2024-03-26T04:17:50.929898+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:50:35.698244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'15","reported_seq":53,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.145487+0000","last_change":"2024-03-26T04:17:13.145335+0000","last_active":"2024-03-26T04:17:51.145487+0000","last_peered":"2024-03-26T04:17:51.145487+0000","last_clean":"2024-03-26T04:17:51.145487+0000","last_became_active":"2024-03-26T04:17:13.144919+0000","last_became_peered":"2024-03-26T04:17:13.144919+0000","last_unstale":"2024-03-26T04:17:51.145487+0000","last_undegraded":"2024-03-26T04:17:51.145487+0000","last_fullsized":"2024-03-26T04:17:51.145487+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:23:37.226689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.330673+0000","last_change":"2024-03-26T04:17:17.258204+0000","last_active":"2024-03-26T04:17:51.330673+0000","last_peered":"2024-03-26T04:17:51.330673+0000","last_clean":"2024-03-26T04:17:51.330673+0000","last_became_active":"2024-03-26T04:17:17.257849+0000","last_became_peered":"2024-03-26T04:17:17.257849+0000","last_unstale":"2024-03-26T04:17:51.330673+0000","last_undegraded":"2024-03-26T04:17:51.330673+0000","last_fullsized":"2024-03-26T04:17:51.330673+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:32:38.220150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.145491+0000","last_change":"2024-03-26T04:17:15.169117+0000","last_active":"2024-03-26T04:17:51.145491+0000","last_peered":"2024-03-26T04:17:51.145491+0000","last_clean":"2024-03-26T04:17:51.145491+0000","last_became_active":"2024-03-26T04:17:15.168745+0000","last_became_peered":"2024-03-26T04:17:15.168745+0000","last_unstale":"2024-03-26T04:17:51.145491+0000","last_undegraded":"2024-03-26T04:17:51.145491+0000","last_fullsized":"2024-03-26T04:17:51.145491+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:30:25.756222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.331278+0000","last_change":"2024-03-26T04:17:11.507897+0000","last_active":"2024-03-26T04:17:51.331278+0000","last_peered":"2024-03-26T04:17:51.331278+0000","last_clean":"2024-03-26T04:17:51.331278+0000","last_became_active":"2024-03-26T04:17:11.507487+0000","last_became_peered":"2024-03-26T04:17:11.507487+0000","last_unstale":"2024-03-26T04:17:51.331278+0000","last_undegraded":"2024-03-26T04:17:51.331278+0000","last_fullsized":"2024-03-26T04:17:51.331278+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:16:56.597175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'39","reported_seq":98,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.330472+0000","last_change":"2024-03-26T04:17:13.154016+0000","last_active":"2024-03-26T04:17:51.330472+0000","last_peered":"2024-03-26T04:17:51.330472+0000","last_clean":"2024-03-26T04:17:51.330472+0000","last_became_active":"2024-03-26T04:17:13.153600+0000","last_became_peered":"2024-03-26T04:17:13.153600+0000","last_unstale":"2024-03-26T04:17:51.330472+0000","last_undegraded":"2024-03-26T04:17:51.330472+0000","last_fullsized":"2024-03-26T04:17:51.330472+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:30:30.806406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.928629+0000","last_change":"2024-03-26T04:17:17.175565+0000","last_active":"2024-03-26T04:17:50.928629+0000","last_peered":"2024-03-26T04:17:50.928629+0000","last_clean":"2024-03-26T04:17:50.928629+0000","last_became_active":"2024-03-26T04:17:17.175116+0000","last_became_peered":"2024-03-26T04:17:17.175116+0000","last_unstale":"2024-03-26T04:17:50.928629+0000","last_undegraded":"2024-03-26T04:17:50.928629+0000","last_fullsized":"2024-03-26T04:17:50.928629+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:42:41.024222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.031867+0000","last_change":"2024-03-26T04:17:15.166332+0000","last_active":"2024-03-26T04:17:51.031867+0000","last_peered":"2024-03-26T04:17:51.031867+0000","last_clean":"2024-03-26T04:17:51.031867+0000","last_became_active":"2024-03-26T04:17:15.166001+0000","last_became_peered":"2024-03-26T04:17:15.166001+0000","last_unstale":"2024-03-26T04:17:51.031867+0000","last_undegraded":"2024-03-26T04:17:51.031867+0000","last_fullsized":"2024-03-26T04:17:51.031867+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:51:07.301584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.032004+0000","last_change":"2024-03-26T04:17:11.136554+0000","last_active":"2024-03-26T04:17:51.032004+0000","last_peered":"2024-03-26T04:17:51.032004+0000","last_clean":"2024-03-26T04:17:51.032004+0000","last_became_active":"2024-03-26T04:17:11.136263+0000","last_became_peered":"2024-03-26T04:17:11.136263+0000","last_unstale":"2024-03-26T04:17:51.032004+0000","last_undegraded":"2024-03-26T04:17:51.032004+0000","last_fullsized":"2024-03-26T04:17:51.032004+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:58:56.300937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'21","reported_seq":65,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.146039+0000","last_change":"2024-03-26T04:17:13.150081+0000","last_active":"2024-03-26T04:17:51.146039+0000","last_peered":"2024-03-26T04:17:51.146039+0000","last_clean":"2024-03-26T04:17:51.146039+0000","last_became_active":"2024-03-26T04:17:13.149623+0000","last_became_peered":"2024-03-26T04:17:13.149623+0000","last_unstale":"2024-03-26T04:17:51.146039+0000","last_undegraded":"2024-03-26T04:17:51.146039+0000","last_fullsized":"2024-03-26T04:17:51.146039+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:12:11.987017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.031618+0000","last_change":"2024-03-26T04:17:17.168717+0000","last_active":"2024-03-26T04:17:51.031618+0000","last_peered":"2024-03-26T04:17:51.031618+0000","last_clean":"2024-03-26T04:17:51.031618+0000","last_became_active":"2024-03-26T04:17:17.168428+0000","last_became_peered":"2024-03-26T04:17:17.168428+0000","last_unstale":"2024-03-26T04:17:51.031618+0000","last_undegraded":"2024-03-26T04:17:51.031618+0000","last_fullsized":"2024-03-26T04:17:51.031618+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:46:43.689601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.931351+0000","last_change":"2024-03-26T04:17:15.172381+0000","last_active":"2024-03-26T04:17:50.931351+0000","last_peered":"2024-03-26T04:17:50.931351+0000","last_clean":"2024-03-26T04:17:50.931351+0000","last_became_active":"2024-03-26T04:17:15.172066+0000","last_became_peered":"2024-03-26T04:17:15.172066+0000","last_unstale":"2024-03-26T04:17:50.931351+0000","last_undegraded":"2024-03-26T04:17:50.931351+0000","last_fullsized":"2024-03-26T04:17:50.931351+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:56:02.948271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.031936+0000","last_change":"2024-03-26T04:17:11.136840+0000","last_active":"2024-03-26T04:17:51.031936+0000","last_peered":"2024-03-26T04:17:51.031936+0000","last_clean":"2024-03-26T04:17:51.031936+0000","last_became_active":"2024-03-26T04:17:11.135863+0000","last_became_peered":"2024-03-26T04:17:11.135863+0000","last_unstale":"2024-03-26T04:17:51.031936+0000","last_undegraded":"2024-03-26T04:17:51.031936+0000","last_fullsized":"2024-03-26T04:17:51.031936+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:43:04.439547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'11","reported_seq":47,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.145915+0000","last_change":"2024-03-26T04:17:13.150181+0000","last_active":"2024-03-26T04:17:51.145915+0000","last_peered":"2024-03-26T04:17:51.145915+0000","last_clean":"2024-03-26T04:17:51.145915+0000","last_became_active":"2024-03-26T04:17:13.149774+0000","last_became_peered":"2024-03-26T04:17:13.149774+0000","last_unstale":"2024-03-26T04:17:51.145915+0000","last_undegraded":"2024-03-26T04:17:51.145915+0000","last_fullsized":"2024-03-26T04:17:51.145915+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:08:47.713336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.146328+0000","last_change":"2024-03-26T04:17:17.256561+0000","last_active":"2024-03-26T04:17:51.146328+0000","last_peered":"2024-03-26T04:17:51.146328+0000","last_clean":"2024-03-26T04:17:51.146328+0000","last_became_active":"2024-03-26T04:17:17.256246+0000","last_became_peered":"2024-03-26T04:17:17.256246+0000","last_unstale":"2024-03-26T04:17:51.146328+0000","last_undegraded":"2024-03-26T04:17:51.146328+0000","last_fullsized":"2024-03-26T04:17:51.146328+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:41:04.652864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.237314+0000","last_change":"2024-03-26T04:17:15.172128+0000","last_active":"2024-03-26T04:17:51.237314+0000","last_peered":"2024-03-26T04:17:51.237314+0000","last_clean":"2024-03-26T04:17:51.237314+0000","last_became_active":"2024-03-26T04:17:15.170737+0000","last_became_peered":"2024-03-26T04:17:15.170737+0000","last_unstale":"2024-03-26T04:17:51.237314+0000","last_undegraded":"2024-03-26T04:17:51.237314+0000","last_fullsized":"2024-03-26T04:17:51.237314+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:06:42.242701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.929682+0000","last_change":"2024-03-26T04:17:11.137150+0000","last_active":"2024-03-26T04:17:50.929682+0000","last_peered":"2024-03-26T04:17:50.929682+0000","last_clean":"2024-03-26T04:17:50.929682+0000","last_became_active":"2024-03-26T04:17:11.136355+0000","last_became_peered":"2024-03-26T04:17:11.136355+0000","last_unstale":"2024-03-26T04:17:50.929682+0000","last_undegraded":"2024-03-26T04:17:50.929682+0000","last_fullsized":"2024-03-26T04:17:50.929682+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:33:13.638639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":50,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.040167+0000","last_change":"2024-03-26T04:17:13.150929+0000","last_active":"2024-03-26T04:17:51.040167+0000","last_peered":"2024-03-26T04:17:51.040167+0000","last_clean":"2024-03-26T04:17:51.040167+0000","last_became_active":"2024-03-26T04:17:13.150574+0000","last_became_peered":"2024-03-26T04:17:13.150574+0000","last_unstale":"2024-03-26T04:17:51.040167+0000","last_undegraded":"2024-03-26T04:17:51.040167+0000","last_fullsized":"2024-03-26T04:17:51.040167+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:11:06.417913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.331272+0000","last_change":"2024-03-26T04:17:17.256539+0000","last_active":"2024-03-26T04:17:51.331272+0000","last_peered":"2024-03-26T04:17:51.331272+0000","last_clean":"2024-03-26T04:17:51.331272+0000","last_became_active":"2024-03-26T04:17:17.256129+0000","last_became_peered":"2024-03-26T04:17:17.256129+0000","last_unstale":"2024-03-26T04:17:51.331272+0000","last_undegraded":"2024-03-26T04:17:51.331272+0000","last_fullsized":"2024-03-26T04:17:51.331272+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:12:09.148071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.145854+0000","last_change":"2024-03-26T04:17:15.169347+0000","last_active":"2024-03-26T04:17:51.145854+0000","last_peered":"2024-03-26T04:17:51.145854+0000","last_clean":"2024-03-26T04:17:51.145854+0000","last_became_active":"2024-03-26T04:17:15.169066+0000","last_became_peered":"2024-03-26T04:17:15.169066+0000","last_unstale":"2024-03-26T04:17:51.145854+0000","last_undegraded":"2024-03-26T04:17:51.145854+0000","last_fullsized":"2024-03-26T04:17:51.145854+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:52:49.722726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"61'1","reported_seq":35,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.146225+0000","last_change":"2024-03-26T04:17:11.507753+0000","last_active":"2024-03-26T04:17:51.146225+0000","last_peered":"2024-03-26T04:17:51.146225+0000","last_clean":"2024-03-26T04:17:51.146225+0000","last_became_active":"2024-03-26T04:17:11.507392+0000","last_became_peered":"2024-03-26T04:17:11.507392+0000","last_unstale":"2024-03-26T04:17:51.146225+0000","last_undegraded":"2024-03-26T04:17:51.146225+0000","last_fullsized":"2024-03-26T04:17:51.146225+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:54:24.277741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'18","reported_seq":60,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.330745+0000","last_change":"2024-03-26T04:17:13.153993+0000","last_active":"2024-03-26T04:17:51.330745+0000","last_peered":"2024-03-26T04:17:51.330745+0000","last_clean":"2024-03-26T04:17:51.330745+0000","last_became_active":"2024-03-26T04:17:13.153672+0000","last_became_peered":"2024-03-26T04:17:13.153672+0000","last_unstale":"2024-03-26T04:17:51.330745+0000","last_undegraded":"2024-03-26T04:17:51.330745+0000","last_fullsized":"2024-03-26T04:17:51.330745+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:14:55.758721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":18,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.931982+0000","last_change":"2024-03-26T04:17:17.175046+0000","last_active":"2024-03-26T04:17:50.931982+0000","last_peered":"2024-03-26T04:17:50.931982+0000","last_clean":"2024-03-26T04:17:50.931982+0000","last_became_active":"2024-03-26T04:17:17.174698+0000","last_became_peered":"2024-03-26T04:17:17.174698+0000","last_unstale":"2024-03-26T04:17:50.931982+0000","last_undegraded":"2024-03-26T04:17:50.931982+0000","last_fullsized":"2024-03-26T04:17:50.931982+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:04:46.045626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"68'8","reported_seq":48,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:18:21.559948+0000","last_change":"2024-03-26T04:17:15.172166+0000","last_active":"2024-03-26T04:18:21.559948+0000","last_peered":"2024-03-26T04:18:21.559948+0000","last_clean":"2024-03-26T04:18:21.559948+0000","last_became_active":"2024-03-26T04:17:15.171824+0000","last_became_peered":"2024-03-26T04:17:15.171824+0000","last_unstale":"2024-03-26T04:18:21.559948+0000","last_undegraded":"2024-03-26T04:18:21.559948+0000","last_fullsized":"2024-03-26T04:18:21.559948+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:33:31.669591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":30,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.239704+0000","last_change":"2024-03-26T04:17:11.140251+0000","last_active":"2024-03-26T04:17:51.239704+0000","last_peered":"2024-03-26T04:17:51.239704+0000","last_clean":"2024-03-26T04:17:51.239704+0000","last_became_active":"2024-03-26T04:17:11.139950+0000","last_became_peered":"2024-03-26T04:17:11.139950+0000","last_unstale":"2024-03-26T04:17:51.239704+0000","last_undegraded":"2024-03-26T04:17:51.239704+0000","last_fullsized":"2024-03-26T04:17:51.239704+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:46:15.986079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'16","reported_seq":52,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.930361+0000","last_change":"2024-03-26T04:17:13.893519+0000","last_active":"2024-03-26T04:17:50.930361+0000","last_peered":"2024-03-26T04:17:50.930361+0000","last_clean":"2024-03-26T04:17:50.930361+0000","last_became_active":"2024-03-26T04:17:13.893215+0000","last_became_peered":"2024-03-26T04:17:13.893215+0000","last_unstale":"2024-03-26T04:17:50.930361+0000","last_undegraded":"2024-03-26T04:17:50.930361+0000","last_fullsized":"2024-03-26T04:17:50.930361+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:53:01.567614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.237489+0000","last_change":"2024-03-26T04:17:17.168789+0000","last_active":"2024-03-26T04:17:51.237489+0000","last_peered":"2024-03-26T04:17:51.237489+0000","last_clean":"2024-03-26T04:17:51.237489+0000","last_became_active":"2024-03-26T04:17:17.168392+0000","last_became_peered":"2024-03-26T04:17:17.168392+0000","last_unstale":"2024-03-26T04:17:51.237489+0000","last_undegraded":"2024-03-26T04:17:51.237489+0000","last_fullsized":"2024-03-26T04:17:51.237489+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:37:24.318835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.930556+0000","last_change":"2024-03-26T04:17:15.167414+0000","last_active":"2024-03-26T04:17:50.930556+0000","last_peered":"2024-03-26T04:17:50.930556+0000","last_clean":"2024-03-26T04:17:50.930556+0000","last_became_active":"2024-03-26T04:17:15.167090+0000","last_became_peered":"2024-03-26T04:17:15.167090+0000","last_unstale":"2024-03-26T04:17:50.930556+0000","last_undegraded":"2024-03-26T04:17:50.930556+0000","last_fullsized":"2024-03-26T04:17:50.930556+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:04:04.380465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.329683+0000","last_change":"2024-03-26T04:17:11.136061+0000","last_active":"2024-03-26T04:17:51.329683+0000","last_peered":"2024-03-26T04:17:51.329683+0000","last_clean":"2024-03-26T04:17:51.329683+0000","last_became_active":"2024-03-26T04:17:11.135732+0000","last_became_peered":"2024-03-26T04:17:11.135732+0000","last_unstale":"2024-03-26T04:17:51.329683+0000","last_undegraded":"2024-03-26T04:17:51.329683+0000","last_fullsized":"2024-03-26T04:17:51.329683+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:48:33.762367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'19","reported_seq":58,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.331088+0000","last_change":"2024-03-26T04:17:13.147788+0000","last_active":"2024-03-26T04:17:51.331088+0000","last_peered":"2024-03-26T04:17:51.331088+0000","last_clean":"2024-03-26T04:17:51.331088+0000","last_became_active":"2024-03-26T04:17:13.147466+0000","last_became_peered":"2024-03-26T04:17:13.147466+0000","last_unstale":"2024-03-26T04:17:51.331088+0000","last_undegraded":"2024-03-26T04:17:51.331088+0000","last_fullsized":"2024-03-26T04:17:51.331088+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:47:37.680465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.931306+0000","last_change":"2024-03-26T04:17:17.174786+0000","last_active":"2024-03-26T04:17:50.931306+0000","last_peered":"2024-03-26T04:17:50.931306+0000","last_clean":"2024-03-26T04:17:50.931306+0000","last_became_active":"2024-03-26T04:17:17.174464+0000","last_became_peered":"2024-03-26T04:17:17.174464+0000","last_unstale":"2024-03-26T04:17:50.931306+0000","last_undegraded":"2024-03-26T04:17:50.931306+0000","last_fullsized":"2024-03-26T04:17:50.931306+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:40:29.421467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.330498+0000","last_change":"2024-03-26T04:17:15.171962+0000","last_active":"2024-03-26T04:17:51.330498+0000","last_peered":"2024-03-26T04:17:51.330498+0000","last_clean":"2024-03-26T04:17:51.330498+0000","last_became_active":"2024-03-26T04:17:15.171669+0000","last_became_peered":"2024-03-26T04:17:15.171669+0000","last_unstale":"2024-03-26T04:17:51.330498+0000","last_undegraded":"2024-03-26T04:17:51.330498+0000","last_fullsized":"2024-03-26T04:17:51.330498+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:33:59.964435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.929796+0000","last_change":"2024-03-26T04:17:11.136085+0000","last_active":"2024-03-26T04:17:50.929796+0000","last_peered":"2024-03-26T04:17:50.929796+0000","last_clean":"2024-03-26T04:17:50.929796+0000","last_became_active":"2024-03-26T04:17:11.135750+0000","last_became_peered":"2024-03-26T04:17:11.135750+0000","last_unstale":"2024-03-26T04:17:50.929796+0000","last_undegraded":"2024-03-26T04:17:50.929796+0000","last_fullsized":"2024-03-26T04:17:50.929796+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:52:57.756684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":31,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.329906+0000","last_change":"2024-03-26T04:17:13.155098+0000","last_active":"2024-03-26T04:17:51.329906+0000","last_peered":"2024-03-26T04:17:51.329906+0000","last_clean":"2024-03-26T04:17:51.329906+0000","last_became_active":"2024-03-26T04:17:13.154586+0000","last_became_peered":"2024-03-26T04:17:13.154586+0000","last_unstale":"2024-03-26T04:17:51.329906+0000","last_undegraded":"2024-03-26T04:17:51.329906+0000","last_fullsized":"2024-03-26T04:17:51.329906+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:09:09.517693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.330649+0000","last_change":"2024-03-26T04:17:17.256626+0000","last_active":"2024-03-26T04:17:51.330649+0000","last_peered":"2024-03-26T04:17:51.330649+0000","last_clean":"2024-03-26T04:17:51.330649+0000","last_became_active":"2024-03-26T04:17:17.256180+0000","last_became_peered":"2024-03-26T04:17:17.256180+0000","last_unstale":"2024-03-26T04:17:51.330649+0000","last_undegraded":"2024-03-26T04:17:51.330649+0000","last_fullsized":"2024-03-26T04:17:51.330649+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:14:04.504206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.931575+0000","last_change":"2024-03-26T04:17:15.174230+0000","last_active":"2024-03-26T04:17:50.931575+0000","last_peered":"2024-03-26T04:17:50.931575+0000","last_clean":"2024-03-26T04:17:50.931575+0000","last_became_active":"2024-03-26T04:17:15.173930+0000","last_became_peered":"2024-03-26T04:17:15.173930+0000","last_unstale":"2024-03-26T04:17:50.931575+0000","last_undegraded":"2024-03-26T04:17:50.931575+0000","last_fullsized":"2024-03-26T04:17:50.931575+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:51:42.310601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.032221+0000","last_change":"2024-03-26T04:17:11.138254+0000","last_active":"2024-03-26T04:17:51.032221+0000","last_peered":"2024-03-26T04:17:51.032221+0000","last_clean":"2024-03-26T04:17:51.032221+0000","last_became_active":"2024-03-26T04:17:11.137908+0000","last_became_peered":"2024-03-26T04:17:11.137908+0000","last_unstale":"2024-03-26T04:17:51.032221+0000","last_undegraded":"2024-03-26T04:17:51.032221+0000","last_fullsized":"2024-03-26T04:17:51.032221+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:25:22.795868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"68'17","reported_seq":56,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.330368+0000","last_change":"2024-03-26T04:17:13.894243+0000","last_active":"2024-03-26T04:17:51.330368+0000","last_peered":"2024-03-26T04:17:51.330368+0000","last_clean":"2024-03-26T04:17:51.330368+0000","last_became_active":"2024-03-26T04:17:13.893923+0000","last_became_peered":"2024-03-26T04:17:13.893923+0000","last_unstale":"2024-03-26T04:17:51.330368+0000","last_undegraded":"2024-03-26T04:17:51.330368+0000","last_fullsized":"2024-03-26T04:17:51.330368+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:48:34.406148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.928367+0000","last_change":"2024-03-26T04:17:17.176148+0000","last_active":"2024-03-26T04:17:50.928367+0000","last_peered":"2024-03-26T04:17:50.928367+0000","last_clean":"2024-03-26T04:17:50.928367+0000","last_became_active":"2024-03-26T04:17:17.175759+0000","last_became_peered":"2024-03-26T04:17:17.175759+0000","last_unstale":"2024-03-26T04:17:50.928367+0000","last_undegraded":"2024-03-26T04:17:50.928367+0000","last_fullsized":"2024-03-26T04:17:50.928367+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:01:17.171602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.330781+0000","last_change":"2024-03-26T04:17:15.174947+0000","last_active":"2024-03-26T04:17:51.330781+0000","last_peered":"2024-03-26T04:17:51.330781+0000","last_clean":"2024-03-26T04:17:51.330781+0000","last_became_active":"2024-03-26T04:17:15.174534+0000","last_became_peered":"2024-03-26T04:17:15.174534+0000","last_unstale":"2024-03-26T04:17:51.330781+0000","last_undegraded":"2024-03-26T04:17:51.330781+0000","last_fullsized":"2024-03-26T04:17:51.330781+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:32:57.104089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.932678+0000","last_change":"2024-03-26T04:17:11.135999+0000","last_active":"2024-03-26T04:17:50.932678+0000","last_peered":"2024-03-26T04:17:50.932678+0000","last_clean":"2024-03-26T04:17:50.932678+0000","last_became_active":"2024-03-26T04:17:11.135609+0000","last_became_peered":"2024-03-26T04:17:11.135609+0000","last_unstale":"2024-03-26T04:17:50.932678+0000","last_undegraded":"2024-03-26T04:17:50.932678+0000","last_fullsized":"2024-03-26T04:17:50.932678+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:32:12.218747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":41,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.240113+0000","last_change":"2024-03-26T04:17:13.144493+0000","last_active":"2024-03-26T04:17:51.240113+0000","last_peered":"2024-03-26T04:17:51.240113+0000","last_clean":"2024-03-26T04:17:51.240113+0000","last_became_active":"2024-03-26T04:17:13.144199+0000","last_became_peered":"2024-03-26T04:17:13.144199+0000","last_unstale":"2024-03-26T04:17:51.240113+0000","last_undegraded":"2024-03-26T04:17:51.240113+0000","last_fullsized":"2024-03-26T04:17:51.240113+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:22:58.859187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.330146+0000","last_change":"2024-03-26T04:17:17.181095+0000","last_active":"2024-03-26T04:17:51.330146+0000","last_peered":"2024-03-26T04:17:51.330146+0000","last_clean":"2024-03-26T04:17:51.330146+0000","last_became_active":"2024-03-26T04:17:17.180753+0000","last_became_peered":"2024-03-26T04:17:17.180753+0000","last_unstale":"2024-03-26T04:17:51.330146+0000","last_undegraded":"2024-03-26T04:17:51.330146+0000","last_fullsized":"2024-03-26T04:17:51.330146+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:34:56.012183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.239899+0000","last_change":"2024-03-26T04:17:15.173684+0000","last_active":"2024-03-26T04:17:51.239899+0000","last_peered":"2024-03-26T04:17:51.239899+0000","last_clean":"2024-03-26T04:17:51.239899+0000","last_became_active":"2024-03-26T04:17:15.173289+0000","last_became_peered":"2024-03-26T04:17:15.173289+0000","last_unstale":"2024-03-26T04:17:51.239899+0000","last_undegraded":"2024-03-26T04:17:51.239899+0000","last_fullsized":"2024-03-26T04:17:51.239899+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:35:13.135620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.145232+0000","last_change":"2024-03-26T04:17:11.140336+0000","last_active":"2024-03-26T04:17:51.145232+0000","last_peered":"2024-03-26T04:17:51.145232+0000","last_clean":"2024-03-26T04:17:51.145232+0000","last_became_active":"2024-03-26T04:17:11.139978+0000","last_became_peered":"2024-03-26T04:17:11.139978+0000","last_unstale":"2024-03-26T04:17:51.145232+0000","last_undegraded":"2024-03-26T04:17:51.145232+0000","last_fullsized":"2024-03-26T04:17:51.145232+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:14:43.523655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":44,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.146199+0000","last_change":"2024-03-26T04:17:13.144107+0000","last_active":"2024-03-26T04:17:51.146199+0000","last_peered":"2024-03-26T04:17:51.146199+0000","last_clean":"2024-03-26T04:17:51.146199+0000","last_became_active":"2024-03-26T04:17:13.143748+0000","last_became_peered":"2024-03-26T04:17:13.143748+0000","last_unstale":"2024-03-26T04:17:51.146199+0000","last_undegraded":"2024-03-26T04:17:51.146199+0000","last_fullsized":"2024-03-26T04:17:51.146199+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:58:01.925951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.930025+0000","last_change":"2024-03-26T04:17:17.179372+0000","last_active":"2024-03-26T04:17:50.930025+0000","last_peered":"2024-03-26T04:17:50.930025+0000","last_clean":"2024-03-26T04:17:50.930025+0000","last_became_active":"2024-03-26T04:17:17.179051+0000","last_became_peered":"2024-03-26T04:17:17.179051+0000","last_unstale":"2024-03-26T04:17:50.930025+0000","last_undegraded":"2024-03-26T04:17:50.930025+0000","last_fullsized":"2024-03-26T04:17:50.930025+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:46:32.062312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":45,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:18:21.559948+0000","last_change":"2024-03-26T04:17:15.169061+0000","last_active":"2024-03-26T04:18:21.559948+0000","last_peered":"2024-03-26T04:18:21.559948+0000","last_clean":"2024-03-26T04:18:21.559948+0000","last_became_active":"2024-03-26T04:17:15.168757+0000","last_became_peered":"2024-03-26T04:17:15.168757+0000","last_unstale":"2024-03-26T04:18:21.559948+0000","last_undegraded":"2024-03-26T04:18:21.559948+0000","last_fullsized":"2024-03-26T04:18:21.559948+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:31:54.756455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.931823+0000","last_change":"2024-03-26T04:17:11.510012+0000","last_active":"2024-03-26T04:17:50.931823+0000","last_peered":"2024-03-26T04:17:50.931823+0000","last_clean":"2024-03-26T04:17:50.931823+0000","last_became_active":"2024-03-26T04:17:11.509770+0000","last_became_peered":"2024-03-26T04:17:11.509770+0000","last_unstale":"2024-03-26T04:17:50.931823+0000","last_undegraded":"2024-03-26T04:17:50.931823+0000","last_fullsized":"2024-03-26T04:17:50.931823+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:23:14.674792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"68'22","reported_seq":64,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.330668+0000","last_change":"2024-03-26T04:17:13.892377+0000","last_active":"2024-03-26T04:17:51.330668+0000","last_peered":"2024-03-26T04:17:51.330668+0000","last_clean":"2024-03-26T04:17:51.330668+0000","last_became_active":"2024-03-26T04:17:13.892011+0000","last_became_peered":"2024-03-26T04:17:13.892011+0000","last_unstale":"2024-03-26T04:17:51.330668+0000","last_undegraded":"2024-03-26T04:17:51.330668+0000","last_fullsized":"2024-03-26T04:17:51.330668+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:46:59.513429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.146560+0000","last_change":"2024-03-26T04:17:17.175008+0000","last_active":"2024-03-26T04:17:51.146560+0000","last_peered":"2024-03-26T04:17:51.146560+0000","last_clean":"2024-03-26T04:17:51.146560+0000","last_became_active":"2024-03-26T04:17:17.174716+0000","last_became_peered":"2024-03-26T04:17:17.174716+0000","last_unstale":"2024-03-26T04:17:51.146560+0000","last_undegraded":"2024-03-26T04:17:51.146560+0000","last_fullsized":"2024-03-26T04:17:51.146560+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:12:14.215487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"68'8","reported_seq":45,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:18:21.559313+0000","last_change":"2024-03-26T04:17:15.167221+0000","last_active":"2024-03-26T04:18:21.559313+0000","last_peered":"2024-03-26T04:18:21.559313+0000","last_clean":"2024-03-26T04:18:21.559313+0000","last_became_active":"2024-03-26T04:17:15.166917+0000","last_became_peered":"2024-03-26T04:17:15.166917+0000","last_unstale":"2024-03-26T04:18:21.559313+0000","last_undegraded":"2024-03-26T04:18:21.559313+0000","last_fullsized":"2024-03-26T04:18:21.559313+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:06:26.208161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.929103+0000","last_change":"2024-03-26T04:17:11.137236+0000","last_active":"2024-03-26T04:17:50.929103+0000","last_peered":"2024-03-26T04:17:50.929103+0000","last_clean":"2024-03-26T04:17:50.929103+0000","last_became_active":"2024-03-26T04:17:11.136939+0000","last_became_peered":"2024-03-26T04:17:11.136939+0000","last_unstale":"2024-03-26T04:17:50.929103+0000","last_undegraded":"2024-03-26T04:17:50.929103+0000","last_fullsized":"2024-03-26T04:17:50.929103+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:25:31.760614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"68'15","reported_seq":53,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.032349+0000","last_change":"2024-03-26T04:17:13.894345+0000","last_active":"2024-03-26T04:17:51.032349+0000","last_peered":"2024-03-26T04:17:51.032349+0000","last_clean":"2024-03-26T04:17:51.032349+0000","last_became_active":"2024-03-26T04:17:13.893928+0000","last_became_peered":"2024-03-26T04:17:13.893928+0000","last_unstale":"2024-03-26T04:17:51.032349+0000","last_undegraded":"2024-03-26T04:17:51.032349+0000","last_fullsized":"2024-03-26T04:17:51.032349+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:51:40.624569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.040322+0000","last_change":"2024-03-26T04:17:17.174273+0000","last_active":"2024-03-26T04:17:51.040322+0000","last_peered":"2024-03-26T04:17:51.040322+0000","last_clean":"2024-03-26T04:17:51.040322+0000","last_became_active":"2024-03-26T04:17:17.173960+0000","last_became_peered":"2024-03-26T04:17:17.173960+0000","last_unstale":"2024-03-26T04:17:51.040322+0000","last_undegraded":"2024-03-26T04:17:51.040322+0000","last_fullsized":"2024-03-26T04:17:51.040322+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:41:27.899074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.331533+0000","last_change":"2024-03-26T04:17:15.166097+0000","last_active":"2024-03-26T04:17:51.331533+0000","last_peered":"2024-03-26T04:17:51.331533+0000","last_clean":"2024-03-26T04:17:51.331533+0000","last_became_active":"2024-03-26T04:17:15.165753+0000","last_became_peered":"2024-03-26T04:17:15.165753+0000","last_unstale":"2024-03-26T04:17:51.331533+0000","last_undegraded":"2024-03-26T04:17:51.331533+0000","last_fullsized":"2024-03-26T04:17:51.331533+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:06:35.354370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.931373+0000","last_change":"2024-03-26T04:17:11.140113+0000","last_active":"2024-03-26T04:17:50.931373+0000","last_peered":"2024-03-26T04:17:50.931373+0000","last_clean":"2024-03-26T04:17:50.931373+0000","last_became_active":"2024-03-26T04:17:11.139843+0000","last_became_peered":"2024-03-26T04:17:11.139843+0000","last_unstale":"2024-03-26T04:17:50.931373+0000","last_undegraded":"2024-03-26T04:17:50.931373+0000","last_fullsized":"2024-03-26T04:17:50.931373+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:45:17.483706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'21","reported_seq":65,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.930435+0000","last_change":"2024-03-26T04:17:13.893466+0000","last_active":"2024-03-26T04:17:50.930435+0000","last_peered":"2024-03-26T04:17:50.930435+0000","last_clean":"2024-03-26T04:17:50.930435+0000","last_became_active":"2024-03-26T04:17:13.893181+0000","last_became_peered":"2024-03-26T04:17:13.893181+0000","last_unstale":"2024-03-26T04:17:50.930435+0000","last_undegraded":"2024-03-26T04:17:50.930435+0000","last_fullsized":"2024-03-26T04:17:50.930435+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:28:35.807341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.932585+0000","last_change":"2024-03-26T04:17:17.258601+0000","last_active":"2024-03-26T04:17:50.932585+0000","last_peered":"2024-03-26T04:17:50.932585+0000","last_clean":"2024-03-26T04:17:50.932585+0000","last_became_active":"2024-03-26T04:17:17.258289+0000","last_became_peered":"2024-03-26T04:17:17.258289+0000","last_unstale":"2024-03-26T04:17:50.932585+0000","last_undegraded":"2024-03-26T04:17:50.932585+0000","last_fullsized":"2024-03-26T04:17:50.932585+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:25:22.120180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.032207+0000","last_change":"2024-03-26T04:17:15.170597+0000","last_active":"2024-03-26T04:17:51.032207+0000","last_peered":"2024-03-26T04:17:51.032207+0000","last_clean":"2024-03-26T04:17:51.032207+0000","last_became_active":"2024-03-26T04:17:15.170275+0000","last_became_peered":"2024-03-26T04:17:15.170275+0000","last_unstale":"2024-03-26T04:17:51.032207+0000","last_undegraded":"2024-03-26T04:17:51.032207+0000","last_fullsized":"2024-03-26T04:17:51.032207+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:24:38.102601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.330046+0000","last_change":"2024-03-26T04:17:11.138327+0000","last_active":"2024-03-26T04:17:51.330046+0000","last_peered":"2024-03-26T04:17:51.330046+0000","last_clean":"2024-03-26T04:17:51.330046+0000","last_became_active":"2024-03-26T04:17:11.138023+0000","last_became_peered":"2024-03-26T04:17:11.138023+0000","last_unstale":"2024-03-26T04:17:51.330046+0000","last_undegraded":"2024-03-26T04:17:51.330046+0000","last_fullsized":"2024-03-26T04:17:51.330046+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:48:15.477574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'12","reported_seq":46,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.331454+0000","last_change":"2024-03-26T04:17:13.892376+0000","last_active":"2024-03-26T04:17:51.331454+0000","last_peered":"2024-03-26T04:17:51.331454+0000","last_clean":"2024-03-26T04:17:51.331454+0000","last_became_active":"2024-03-26T04:17:13.892026+0000","last_became_peered":"2024-03-26T04:17:13.892026+0000","last_unstale":"2024-03-26T04:17:51.331454+0000","last_undegraded":"2024-03-26T04:17:51.331454+0000","last_fullsized":"2024-03-26T04:17:51.331454+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:45:38.509390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.031973+0000","last_change":"2024-03-26T04:17:17.256646+0000","last_active":"2024-03-26T04:17:51.031973+0000","last_peered":"2024-03-26T04:17:51.031973+0000","last_clean":"2024-03-26T04:17:51.031973+0000","last_became_active":"2024-03-26T04:17:17.256134+0000","last_became_peered":"2024-03-26T04:17:17.256134+0000","last_unstale":"2024-03-26T04:17:51.031973+0000","last_undegraded":"2024-03-26T04:17:51.031973+0000","last_fullsized":"2024-03-26T04:17:51.031973+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:38:49.190015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":41,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.932450+0000","last_change":"2024-03-26T04:17:11.142324+0000","last_active":"2024-03-26T04:17:50.932450+0000","last_peered":"2024-03-26T04:17:50.932450+0000","last_clean":"2024-03-26T04:17:50.932450+0000","last_became_active":"2024-03-26T04:17:11.142005+0000","last_became_peered":"2024-03-26T04:17:11.142005+0000","last_unstale":"2024-03-26T04:17:50.932450+0000","last_undegraded":"2024-03-26T04:17:50.932450+0000","last_fullsized":"2024-03-26T04:17:50.932450+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:43:35.639879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"68'27","reported_seq":74,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.031638+0000","last_change":"2024-03-26T04:17:13.894492+0000","last_active":"2024-03-26T04:17:51.031638+0000","last_peered":"2024-03-26T04:17:51.031638+0000","last_clean":"2024-03-26T04:17:51.031638+0000","last_became_active":"2024-03-26T04:17:13.893841+0000","last_became_peered":"2024-03-26T04:17:13.893841+0000","last_unstale":"2024-03-26T04:17:51.031638+0000","last_undegraded":"2024-03-26T04:17:51.031638+0000","last_fullsized":"2024-03-26T04:17:51.031638+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:40:39.438254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"68'8","reported_seq":48,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:18:21.559948+0000","last_change":"2024-03-26T04:17:15.172117+0000","last_active":"2024-03-26T04:18:21.559948+0000","last_peered":"2024-03-26T04:18:21.559948+0000","last_clean":"2024-03-26T04:18:21.559948+0000","last_became_active":"2024-03-26T04:17:15.171730+0000","last_became_peered":"2024-03-26T04:17:15.171730+0000","last_unstale":"2024-03-26T04:18:21.559948+0000","last_undegraded":"2024-03-26T04:18:21.559948+0000","last_fullsized":"2024-03-26T04:18:21.559948+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:26:00.568738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.330568+0000","last_change":"2024-03-26T04:17:17.177097+0000","last_active":"2024-03-26T04:17:51.330568+0000","last_peered":"2024-03-26T04:17:51.330568+0000","last_clean":"2024-03-26T04:17:51.330568+0000","last_became_active":"2024-03-26T04:17:17.176333+0000","last_became_peered":"2024-03-26T04:17:17.176333+0000","last_unstale":"2024-03-26T04:17:51.330568+0000","last_undegraded":"2024-03-26T04:17:51.330568+0000","last_fullsized":"2024-03-26T04:17:51.330568+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:53:08.534283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.931922+0000","last_change":"2024-03-26T04:17:11.143613+0000","last_active":"2024-03-26T04:17:50.931922+0000","last_peered":"2024-03-26T04:17:50.931922+0000","last_clean":"2024-03-26T04:17:50.931922+0000","last_became_active":"2024-03-26T04:17:11.143279+0000","last_became_peered":"2024-03-26T04:17:11.143279+0000","last_unstale":"2024-03-26T04:17:50.931922+0000","last_undegraded":"2024-03-26T04:17:50.931922+0000","last_fullsized":"2024-03-26T04:17:50.931922+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:23:16.431330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"68'24","reported_seq":72,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.145861+0000","last_change":"2024-03-26T04:17:13.150344+0000","last_active":"2024-03-26T04:17:51.145861+0000","last_peered":"2024-03-26T04:17:51.145861+0000","last_clean":"2024-03-26T04:17:51.145861+0000","last_became_active":"2024-03-26T04:17:13.149870+0000","last_became_peered":"2024-03-26T04:17:13.149870+0000","last_unstale":"2024-03-26T04:17:51.145861+0000","last_undegraded":"2024-03-26T04:17:51.145861+0000","last_fullsized":"2024-03-26T04:17:51.145861+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:19:52.903517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.039732+0000","last_change":"2024-03-26T04:17:15.165655+0000","last_active":"2024-03-26T04:17:51.039732+0000","last_peered":"2024-03-26T04:17:51.039732+0000","last_clean":"2024-03-26T04:17:51.039732+0000","last_became_active":"2024-03-26T04:17:15.165308+0000","last_became_peered":"2024-03-26T04:17:15.165308+0000","last_unstale":"2024-03-26T04:17:51.039732+0000","last_undegraded":"2024-03-26T04:17:51.039732+0000","last_fullsized":"2024-03-26T04:17:51.039732+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:42:38.061055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.931167+0000","last_change":"2024-03-26T04:17:17.176227+0000","last_active":"2024-03-26T04:17:50.931167+0000","last_peered":"2024-03-26T04:17:50.931167+0000","last_clean":"2024-03-26T04:17:50.931167+0000","last_became_active":"2024-03-26T04:17:17.175814+0000","last_became_peered":"2024-03-26T04:17:17.175814+0000","last_unstale":"2024-03-26T04:17:50.931167+0000","last_undegraded":"2024-03-26T04:17:50.931167+0000","last_fullsized":"2024-03-26T04:17:50.931167+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:22:47.174732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.040082+0000","last_change":"2024-03-26T04:17:15.170181+0000","last_active":"2024-03-26T04:17:51.040082+0000","last_peered":"2024-03-26T04:17:51.040082+0000","last_clean":"2024-03-26T04:17:51.040082+0000","last_became_active":"2024-03-26T04:17:15.169851+0000","last_became_peered":"2024-03-26T04:17:15.169851+0000","last_unstale":"2024-03-26T04:17:51.040082+0000","last_undegraded":"2024-03-26T04:17:51.040082+0000","last_fullsized":"2024-03-26T04:17:51.040082+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:47:48.162297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.932892+0000","last_change":"2024-03-26T04:17:11.140098+0000","last_active":"2024-03-26T04:17:50.932892+0000","last_peered":"2024-03-26T04:17:50.932892+0000","last_clean":"2024-03-26T04:17:50.932892+0000","last_became_active":"2024-03-26T04:17:11.139402+0000","last_became_peered":"2024-03-26T04:17:11.139402+0000","last_unstale":"2024-03-26T04:17:50.932892+0000","last_undegraded":"2024-03-26T04:17:50.932892+0000","last_fullsized":"2024-03-26T04:17:50.932892+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:38:39.696009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"68'25","reported_seq":68,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.330718+0000","last_change":"2024-03-26T04:17:13.892572+0000","last_active":"2024-03-26T04:17:51.330718+0000","last_peered":"2024-03-26T04:17:51.330718+0000","last_clean":"2024-03-26T04:17:51.330718+0000","last_became_active":"2024-03-26T04:17:13.892212+0000","last_became_peered":"2024-03-26T04:17:13.892212+0000","last_unstale":"2024-03-26T04:17:51.330718+0000","last_undegraded":"2024-03-26T04:17:51.330718+0000","last_fullsized":"2024-03-26T04:17:51.330718+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:50:49.538879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.928651+0000","last_change":"2024-03-26T04:17:17.176526+0000","last_active":"2024-03-26T04:17:50.928651+0000","last_peered":"2024-03-26T04:17:50.928651+0000","last_clean":"2024-03-26T04:17:50.928651+0000","last_became_active":"2024-03-26T04:17:17.176224+0000","last_became_peered":"2024-03-26T04:17:17.176224+0000","last_unstale":"2024-03-26T04:17:50.928651+0000","last_undegraded":"2024-03-26T04:17:50.928651+0000","last_fullsized":"2024-03-26T04:17:50.928651+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:56:50.803919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.039988+0000","last_change":"2024-03-26T04:17:15.168756+0000","last_active":"2024-03-26T04:17:51.039988+0000","last_peered":"2024-03-26T04:17:51.039988+0000","last_clean":"2024-03-26T04:17:51.039988+0000","last_became_active":"2024-03-26T04:17:15.168402+0000","last_became_peered":"2024-03-26T04:17:15.168402+0000","last_unstale":"2024-03-26T04:17:51.039988+0000","last_undegraded":"2024-03-26T04:17:51.039988+0000","last_fullsized":"2024-03-26T04:17:51.039988+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:55:28.119084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.032026+0000","last_change":"2024-03-26T04:17:11.137375+0000","last_active":"2024-03-26T04:17:51.032026+0000","last_peered":"2024-03-26T04:17:51.032026+0000","last_clean":"2024-03-26T04:17:51.032026+0000","last_became_active":"2024-03-26T04:17:11.137073+0000","last_became_peered":"2024-03-26T04:17:11.137073+0000","last_unstale":"2024-03-26T04:17:51.032026+0000","last_undegraded":"2024-03-26T04:17:51.032026+0000","last_fullsized":"2024-03-26T04:17:51.032026+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:10:09.759028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"68'15","reported_seq":53,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.929501+0000","last_change":"2024-03-26T04:17:13.144023+0000","last_active":"2024-03-26T04:17:50.929501+0000","last_peered":"2024-03-26T04:17:50.929501+0000","last_clean":"2024-03-26T04:17:50.929501+0000","last_became_active":"2024-03-26T04:17:13.143687+0000","last_became_peered":"2024-03-26T04:17:13.143687+0000","last_unstale":"2024-03-26T04:17:50.929501+0000","last_undegraded":"2024-03-26T04:17:50.929501+0000","last_fullsized":"2024-03-26T04:17:50.929501+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:00:31.849325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.330978+0000","last_change":"2024-03-26T04:17:17.177350+0000","last_active":"2024-03-26T04:17:51.330978+0000","last_peered":"2024-03-26T04:17:51.330978+0000","last_clean":"2024-03-26T04:17:51.330978+0000","last_became_active":"2024-03-26T04:17:17.177019+0000","last_became_peered":"2024-03-26T04:17:17.177019+0000","last_unstale":"2024-03-26T04:17:51.330978+0000","last_undegraded":"2024-03-26T04:17:51.330978+0000","last_fullsized":"2024-03-26T04:17:51.330978+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:07:32.652282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":40,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.031773+0000","last_change":"2024-03-26T04:17:14.155484+0000","last_active":"2024-03-26T04:17:51.031773+0000","last_peered":"2024-03-26T04:17:51.031773+0000","last_clean":"2024-03-26T04:17:51.031773+0000","last_became_active":"2024-03-26T04:17:10.129516+0000","last_became_peered":"2024-03-26T04:17:10.129516+0000","last_unstale":"2024-03-26T04:17:51.031773+0000","last_undegraded":"2024-03-26T04:17:51.031773+0000","last_fullsized":"2024-03-26T04:17:51.031773+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:09.109486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:09.109486+0000","last_clean_scrub_stamp":"2024-03-26T04:17:09.109486+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:09:47.099491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015728389999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.145107+0000","last_change":"2024-03-26T04:17:11.137658+0000","last_active":"2024-03-26T04:17:51.145107+0000","last_peered":"2024-03-26T04:17:51.145107+0000","last_clean":"2024-03-26T04:17:51.145107+0000","last_became_active":"2024-03-26T04:17:11.137368+0000","last_became_peered":"2024-03-26T04:17:11.137368+0000","last_unstale":"2024-03-26T04:17:51.145107+0000","last_undegraded":"2024-03-26T04:17:51.145107+0000","last_fullsized":"2024-03-26T04:17:51.145107+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:01:39.556213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"68'34","reported_seq":101,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:18:21.559273+0000","last_change":"2024-03-26T04:17:13.154441+0000","last_active":"2024-03-26T04:18:21.559273+0000","last_peered":"2024-03-26T04:18:21.559273+0000","last_clean":"2024-03-26T04:18:21.559273+0000","last_became_active":"2024-03-26T04:17:13.154106+0000","last_became_peered":"2024-03-26T04:17:13.154106+0000","last_unstale":"2024-03-26T04:18:21.559273+0000","last_undegraded":"2024-03-26T04:18:21.559273+0000","last_fullsized":"2024-03-26T04:18:21.559273+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:08:27.442392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.929449+0000","last_change":"2024-03-26T04:17:15.169344+0000","last_active":"2024-03-26T04:17:50.929449+0000","last_peered":"2024-03-26T04:17:50.929449+0000","last_clean":"2024-03-26T04:17:50.929449+0000","last_became_active":"2024-03-26T04:17:15.168809+0000","last_became_peered":"2024-03-26T04:17:15.168809+0000","last_unstale":"2024-03-26T04:17:50.929449+0000","last_undegraded":"2024-03-26T04:17:50.929449+0000","last_fullsized":"2024-03-26T04:17:50.929449+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:21:03.151162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.933037+0000","last_change":"2024-03-26T04:17:17.257393+0000","last_active":"2024-03-26T04:17:50.933037+0000","last_peered":"2024-03-26T04:17:50.933037+0000","last_clean":"2024-03-26T04:17:50.933037+0000","last_became_active":"2024-03-26T04:17:17.257078+0000","last_became_peered":"2024-03-26T04:17:17.257078+0000","last_unstale":"2024-03-26T04:17:50.933037+0000","last_undegraded":"2024-03-26T04:17:50.933037+0000","last_fullsized":"2024-03-26T04:17:50.933037+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:28:06.403543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.240493+0000","last_change":"2024-03-26T04:17:11.507506+0000","last_active":"2024-03-26T04:17:51.240493+0000","last_peered":"2024-03-26T04:17:51.240493+0000","last_clean":"2024-03-26T04:17:51.240493+0000","last_became_active":"2024-03-26T04:17:11.507085+0000","last_became_peered":"2024-03-26T04:17:11.507085+0000","last_unstale":"2024-03-26T04:17:51.240493+0000","last_undegraded":"2024-03-26T04:17:51.240493+0000","last_fullsized":"2024-03-26T04:17:51.240493+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:41:09.015342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":38,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.039947+0000","last_change":"2024-03-26T04:17:14.146806+0000","last_active":"2024-03-26T04:17:51.039947+0000","last_peered":"2024-03-26T04:17:51.039947+0000","last_clean":"2024-03-26T04:17:51.039947+0000","last_became_active":"2024-03-26T04:17:10.130110+0000","last_became_peered":"2024-03-26T04:17:10.130110+0000","last_unstale":"2024-03-26T04:17:51.039947+0000","last_undegraded":"2024-03-26T04:17:51.039947+0000","last_fullsized":"2024-03-26T04:17:51.039947+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:09.109486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:09.109486+0000","last_clean_scrub_stamp":"2024-03-26T04:17:09.109486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:56:41.645772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051742100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"68'19","reported_seq":64,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.237837+0000","last_change":"2024-03-26T04:17:13.145869+0000","last_active":"2024-03-26T04:17:51.237837+0000","last_peered":"2024-03-26T04:17:51.237837+0000","last_clean":"2024-03-26T04:17:51.237837+0000","last_became_active":"2024-03-26T04:17:13.144709+0000","last_became_peered":"2024-03-26T04:17:13.144709+0000","last_unstale":"2024-03-26T04:17:51.237837+0000","last_undegraded":"2024-03-26T04:17:51.237837+0000","last_fullsized":"2024-03-26T04:17:51.237837+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:12:58.044252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.040400+0000","last_change":"2024-03-26T04:17:15.171150+0000","last_active":"2024-03-26T04:17:51.040400+0000","last_peered":"2024-03-26T04:17:51.040400+0000","last_clean":"2024-03-26T04:17:51.040400+0000","last_became_active":"2024-03-26T04:17:15.170576+0000","last_became_peered":"2024-03-26T04:17:15.170576+0000","last_unstale":"2024-03-26T04:17:51.040400+0000","last_undegraded":"2024-03-26T04:17:51.040400+0000","last_fullsized":"2024-03-26T04:17:51.040400+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:20:01.340563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.032225+0000","last_change":"2024-03-26T04:17:17.168363+0000","last_active":"2024-03-26T04:17:51.032225+0000","last_peered":"2024-03-26T04:17:51.032225+0000","last_clean":"2024-03-26T04:17:51.032225+0000","last_became_active":"2024-03-26T04:17:17.168095+0000","last_became_peered":"2024-03-26T04:17:17.168095+0000","last_unstale":"2024-03-26T04:17:51.032225+0000","last_undegraded":"2024-03-26T04:17:51.032225+0000","last_fullsized":"2024-03-26T04:17:51.032225+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:34:15.154296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"70'128","reported_seq":273,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:53.022931+0000","last_change":"2024-03-26T04:17:01.279536+0000","last_active":"2024-03-26T04:17:53.022931+0000","last_peered":"2024-03-26T04:17:53.022931+0000","last_clean":"2024-03-26T04:17:53.022931+0000","last_became_active":"2024-03-26T04:17:01.279151+0000","last_became_peered":"2024-03-26T04:17:01.279151+0000","last_unstale":"2024-03-26T04:17:53.022931+0000","last_undegraded":"2024-03-26T04:17:53.022931+0000","last_fullsized":"2024-03-26T04:17:53.022931+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:14:58.099724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:14:58.099724+0000","last_clean_scrub_stamp":"2024-03-26T04:14:58.099724+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:21:39.312605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.329926+0000","last_change":"2024-03-26T04:17:11.138651+0000","last_active":"2024-03-26T04:17:51.329926+0000","last_peered":"2024-03-26T04:17:51.329926+0000","last_clean":"2024-03-26T04:17:51.329926+0000","last_became_active":"2024-03-26T04:17:11.138304+0000","last_became_peered":"2024-03-26T04:17:11.138304+0000","last_unstale":"2024-03-26T04:17:51.329926+0000","last_undegraded":"2024-03-26T04:17:51.329926+0000","last_fullsized":"2024-03-26T04:17:51.329926+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:48:47.763631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"68'16","reported_seq":63,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:18:21.560202+0000","last_change":"2024-03-26T04:17:13.893740+0000","last_active":"2024-03-26T04:18:21.560202+0000","last_peered":"2024-03-26T04:18:21.560202+0000","last_clean":"2024-03-26T04:18:21.560202+0000","last_became_active":"2024-03-26T04:17:13.893175+0000","last_became_peered":"2024-03-26T04:17:13.893175+0000","last_unstale":"2024-03-26T04:18:21.560202+0000","last_undegraded":"2024-03-26T04:18:21.560202+0000","last_fullsized":"2024-03-26T04:18:21.560202+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:08:28.719073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.932285+0000","last_change":"2024-03-26T04:17:15.167344+0000","last_active":"2024-03-26T04:17:50.932285+0000","last_peered":"2024-03-26T04:17:50.932285+0000","last_clean":"2024-03-26T04:17:50.932285+0000","last_became_active":"2024-03-26T04:17:15.167045+0000","last_became_peered":"2024-03-26T04:17:15.167045+0000","last_unstale":"2024-03-26T04:17:50.932285+0000","last_undegraded":"2024-03-26T04:17:50.932285+0000","last_fullsized":"2024-03-26T04:17:50.932285+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:50:27.618263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"68'1","reported_seq":18,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.241035+0000","last_change":"2024-03-26T04:17:17.170240+0000","last_active":"2024-03-26T04:17:51.241035+0000","last_peered":"2024-03-26T04:17:51.241035+0000","last_clean":"2024-03-26T04:17:51.241035+0000","last_became_active":"2024-03-26T04:17:17.169919+0000","last_became_peered":"2024-03-26T04:17:17.169919+0000","last_unstale":"2024-03-26T04:17:51.241035+0000","last_undegraded":"2024-03-26T04:17:51.241035+0000","last_fullsized":"2024-03-26T04:17:51.241035+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:04:03.612791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.928875+0000","last_change":"2024-03-26T04:17:11.135087+0000","last_active":"2024-03-26T04:17:50.928875+0000","last_peered":"2024-03-26T04:17:50.928875+0000","last_clean":"2024-03-26T04:17:50.928875+0000","last_became_active":"2024-03-26T04:17:11.134742+0000","last_became_peered":"2024-03-26T04:17:11.134742+0000","last_unstale":"2024-03-26T04:17:50.928875+0000","last_undegraded":"2024-03-26T04:17:50.928875+0000","last_fullsized":"2024-03-26T04:17:50.928875+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:58:33.967014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":110,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:18:25.789461+0000","last_change":"2024-03-26T04:17:14.157838+0000","last_active":"2024-03-26T04:18:25.789461+0000","last_peered":"2024-03-26T04:18:25.789461+0000","last_clean":"2024-03-26T04:18:25.789461+0000","last_became_active":"2024-03-26T04:17:10.132378+0000","last_became_peered":"2024-03-26T04:17:10.132378+0000","last_unstale":"2024-03-26T04:18:25.789461+0000","last_undegraded":"2024-03-26T04:18:25.789461+0000","last_fullsized":"2024-03-26T04:18:25.789461+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:09.109486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:09.109486+0000","last_clean_scrub_stamp":"2024-03-26T04:17:09.109486+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:25:16.913920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022493809999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":69,"num_read_kb":64,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":43,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.930119+0000","last_change":"2024-03-26T04:17:13.146942+0000","last_active":"2024-03-26T04:17:50.930119+0000","last_peered":"2024-03-26T04:17:50.930119+0000","last_clean":"2024-03-26T04:17:50.930119+0000","last_became_active":"2024-03-26T04:17:13.146602+0000","last_became_peered":"2024-03-26T04:17:13.146602+0000","last_unstale":"2024-03-26T04:17:50.930119+0000","last_undegraded":"2024-03-26T04:17:50.930119+0000","last_fullsized":"2024-03-26T04:17:50.930119+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:02:06.164330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.032466+0000","last_change":"2024-03-26T04:17:15.169662+0000","last_active":"2024-03-26T04:17:51.032466+0000","last_peered":"2024-03-26T04:17:51.032466+0000","last_clean":"2024-03-26T04:17:51.032466+0000","last_became_active":"2024-03-26T04:17:15.169284+0000","last_became_peered":"2024-03-26T04:17:15.169284+0000","last_unstale":"2024-03-26T04:17:51.032466+0000","last_undegraded":"2024-03-26T04:17:51.032466+0000","last_fullsized":"2024-03-26T04:17:51.032466+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:22:34.956409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.239503+0000","last_change":"2024-03-26T04:17:17.257336+0000","last_active":"2024-03-26T04:17:51.239503+0000","last_peered":"2024-03-26T04:17:51.239503+0000","last_clean":"2024-03-26T04:17:51.239503+0000","last_became_active":"2024-03-26T04:17:17.256133+0000","last_became_peered":"2024-03-26T04:17:17.256133+0000","last_unstale":"2024-03-26T04:17:51.239503+0000","last_undegraded":"2024-03-26T04:17:51.239503+0000","last_fullsized":"2024-03-26T04:17:51.239503+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:10:09.000458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.241247+0000","last_change":"2024-03-26T04:17:11.507607+0000","last_active":"2024-03-26T04:17:51.241247+0000","last_peered":"2024-03-26T04:17:51.241247+0000","last_clean":"2024-03-26T04:17:51.241247+0000","last_became_active":"2024-03-26T04:17:11.507267+0000","last_became_peered":"2024-03-26T04:17:11.507267+0000","last_unstale":"2024-03-26T04:17:51.241247+0000","last_undegraded":"2024-03-26T04:17:51.241247+0000","last_fullsized":"2024-03-26T04:17:51.241247+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:18:02.826317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"68'25","reported_seq":73,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.929776+0000","last_change":"2024-03-26T04:17:13.892595+0000","last_active":"2024-03-26T04:17:50.929776+0000","last_peered":"2024-03-26T04:17:50.929776+0000","last_clean":"2024-03-26T04:17:50.929776+0000","last_became_active":"2024-03-26T04:17:13.892263+0000","last_became_peered":"2024-03-26T04:17:13.892263+0000","last_unstale":"2024-03-26T04:17:50.929776+0000","last_undegraded":"2024-03-26T04:17:50.929776+0000","last_fullsized":"2024-03-26T04:17:50.929776+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:43:22.950743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.330728+0000","last_change":"2024-03-26T04:17:15.167661+0000","last_active":"2024-03-26T04:17:51.330728+0000","last_peered":"2024-03-26T04:17:51.330728+0000","last_clean":"2024-03-26T04:17:51.330728+0000","last_became_active":"2024-03-26T04:17:15.167369+0000","last_became_peered":"2024-03-26T04:17:15.167369+0000","last_unstale":"2024-03-26T04:17:51.330728+0000","last_undegraded":"2024-03-26T04:17:51.330728+0000","last_fullsized":"2024-03-26T04:17:51.330728+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:13:55.556911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.928368+0000","last_change":"2024-03-26T04:17:17.180260+0000","last_active":"2024-03-26T04:17:50.928368+0000","last_peered":"2024-03-26T04:17:50.928368+0000","last_clean":"2024-03-26T04:17:50.928368+0000","last_became_active":"2024-03-26T04:17:17.179927+0000","last_became_peered":"2024-03-26T04:17:17.179927+0000","last_unstale":"2024-03-26T04:17:50.928368+0000","last_undegraded":"2024-03-26T04:17:50.928368+0000","last_fullsized":"2024-03-26T04:17:50.928368+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:48:29.157699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.032199+0000","last_change":"2024-03-26T04:17:11.507753+0000","last_active":"2024-03-26T04:17:51.032199+0000","last_peered":"2024-03-26T04:17:51.032199+0000","last_clean":"2024-03-26T04:17:51.032199+0000","last_became_active":"2024-03-26T04:17:11.507399+0000","last_became_peered":"2024-03-26T04:17:11.507399+0000","last_unstale":"2024-03-26T04:17:51.032199+0000","last_undegraded":"2024-03-26T04:17:51.032199+0000","last_fullsized":"2024-03-26T04:17:51.032199+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:30:25.517263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"68'16","reported_seq":52,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.239985+0000","last_change":"2024-03-26T04:17:13.140889+0000","last_active":"2024-03-26T04:17:51.239985+0000","last_peered":"2024-03-26T04:17:51.239985+0000","last_clean":"2024-03-26T04:17:51.239985+0000","last_became_active":"2024-03-26T04:17:13.140546+0000","last_became_peered":"2024-03-26T04:17:13.140546+0000","last_unstale":"2024-03-26T04:17:51.239985+0000","last_undegraded":"2024-03-26T04:17:51.239985+0000","last_fullsized":"2024-03-26T04:17:51.239985+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:49:44.183611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"68'8","reported_seq":45,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:18:21.559318+0000","last_change":"2024-03-26T04:17:15.168504+0000","last_active":"2024-03-26T04:18:21.559318+0000","last_peered":"2024-03-26T04:18:21.559318+0000","last_clean":"2024-03-26T04:18:21.559318+0000","last_became_active":"2024-03-26T04:17:15.167301+0000","last_became_peered":"2024-03-26T04:17:15.167301+0000","last_unstale":"2024-03-26T04:18:21.559318+0000","last_undegraded":"2024-03-26T04:18:21.559318+0000","last_fullsized":"2024-03-26T04:18:21.559318+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:25:27.460165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.932060+0000","last_change":"2024-03-26T04:17:17.258699+0000","last_active":"2024-03-26T04:17:50.932060+0000","last_peered":"2024-03-26T04:17:50.932060+0000","last_clean":"2024-03-26T04:17:50.932060+0000","last_became_active":"2024-03-26T04:17:17.258405+0000","last_became_peered":"2024-03-26T04:17:17.258405+0000","last_unstale":"2024-03-26T04:17:50.932060+0000","last_undegraded":"2024-03-26T04:17:50.932060+0000","last_fullsized":"2024-03-26T04:17:50.932060+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:45:12.145413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":30,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.930256+0000","last_change":"2024-03-26T04:17:11.134146+0000","last_active":"2024-03-26T04:17:50.930256+0000","last_peered":"2024-03-26T04:17:50.930256+0000","last_clean":"2024-03-26T04:17:50.930256+0000","last_became_active":"2024-03-26T04:17:11.133861+0000","last_became_peered":"2024-03-26T04:17:11.133861+0000","last_unstale":"2024-03-26T04:17:50.930256+0000","last_undegraded":"2024-03-26T04:17:50.930256+0000","last_fullsized":"2024-03-26T04:17:50.930256+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:32:03.317648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"68'20","reported_seq":58,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.145758+0000","last_change":"2024-03-26T04:17:13.142252+0000","last_active":"2024-03-26T04:17:51.145758+0000","last_peered":"2024-03-26T04:17:51.145758+0000","last_clean":"2024-03-26T04:17:51.145758+0000","last_became_active":"2024-03-26T04:17:13.141826+0000","last_became_peered":"2024-03-26T04:17:13.141826+0000","last_unstale":"2024-03-26T04:17:51.145758+0000","last_undegraded":"2024-03-26T04:17:51.145758+0000","last_fullsized":"2024-03-26T04:17:51.145758+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:15:16.751157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":45,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:18:21.559285+0000","last_change":"2024-03-26T04:17:15.166276+0000","last_active":"2024-03-26T04:18:21.559285+0000","last_peered":"2024-03-26T04:18:21.559285+0000","last_clean":"2024-03-26T04:18:21.559285+0000","last_became_active":"2024-03-26T04:17:15.164738+0000","last_became_peered":"2024-03-26T04:17:15.164738+0000","last_unstale":"2024-03-26T04:18:21.559285+0000","last_undegraded":"2024-03-26T04:18:21.559285+0000","last_fullsized":"2024-03-26T04:18:21.559285+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:11:35.075712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.145875+0000","last_change":"2024-03-26T04:17:17.174331+0000","last_active":"2024-03-26T04:17:51.145875+0000","last_peered":"2024-03-26T04:17:51.145875+0000","last_clean":"2024-03-26T04:17:51.145875+0000","last_became_active":"2024-03-26T04:17:17.173982+0000","last_became_peered":"2024-03-26T04:17:17.173982+0000","last_unstale":"2024-03-26T04:17:51.145875+0000","last_undegraded":"2024-03-26T04:17:51.145875+0000","last_fullsized":"2024-03-26T04:17:51.145875+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:31:25.172307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.330046+0000","last_change":"2024-03-26T04:17:11.137412+0000","last_active":"2024-03-26T04:17:51.330046+0000","last_peered":"2024-03-26T04:17:51.330046+0000","last_clean":"2024-03-26T04:17:51.330046+0000","last_became_active":"2024-03-26T04:17:11.137113+0000","last_became_peered":"2024-03-26T04:17:11.137113+0000","last_unstale":"2024-03-26T04:17:51.330046+0000","last_undegraded":"2024-03-26T04:17:51.330046+0000","last_fullsized":"2024-03-26T04:17:51.330046+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:20:59.116143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"68'30","reported_seq":81,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.329965+0000","last_change":"2024-03-26T04:17:13.894120+0000","last_active":"2024-03-26T04:17:51.329965+0000","last_peered":"2024-03-26T04:17:51.329965+0000","last_clean":"2024-03-26T04:17:51.329965+0000","last_became_active":"2024-03-26T04:17:13.893795+0000","last_became_peered":"2024-03-26T04:17:13.893795+0000","last_unstale":"2024-03-26T04:17:51.329965+0000","last_undegraded":"2024-03-26T04:17:51.329965+0000","last_fullsized":"2024-03-26T04:17:51.329965+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:36:09.783064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.145603+0000","last_change":"2024-03-26T04:17:15.170379+0000","last_active":"2024-03-26T04:17:51.145603+0000","last_peered":"2024-03-26T04:17:51.145603+0000","last_clean":"2024-03-26T04:17:51.145603+0000","last_became_active":"2024-03-26T04:17:15.170012+0000","last_became_peered":"2024-03-26T04:17:15.170012+0000","last_unstale":"2024-03-26T04:17:51.145603+0000","last_undegraded":"2024-03-26T04:17:51.145603+0000","last_fullsized":"2024-03-26T04:17:51.145603+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:49:21.109132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.032037+0000","last_change":"2024-03-26T04:17:17.175942+0000","last_active":"2024-03-26T04:17:51.032037+0000","last_peered":"2024-03-26T04:17:51.032037+0000","last_clean":"2024-03-26T04:17:51.032037+0000","last_became_active":"2024-03-26T04:17:17.175378+0000","last_became_peered":"2024-03-26T04:17:17.175378+0000","last_unstale":"2024-03-26T04:17:51.032037+0000","last_undegraded":"2024-03-26T04:17:51.032037+0000","last_fullsized":"2024-03-26T04:17:51.032037+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:53:26.000527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":45,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:18:21.559948+0000","last_change":"2024-03-26T04:17:15.169106+0000","last_active":"2024-03-26T04:18:21.559948+0000","last_peered":"2024-03-26T04:18:21.559948+0000","last_clean":"2024-03-26T04:18:21.559948+0000","last_became_active":"2024-03-26T04:17:15.168175+0000","last_became_peered":"2024-03-26T04:17:15.168175+0000","last_unstale":"2024-03-26T04:18:21.559948+0000","last_undegraded":"2024-03-26T04:18:21.559948+0000","last_fullsized":"2024-03-26T04:18:21.559948+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:14:38.599782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.329674+0000","last_change":"2024-03-26T04:17:11.139149+0000","last_active":"2024-03-26T04:17:51.329674+0000","last_peered":"2024-03-26T04:17:51.329674+0000","last_clean":"2024-03-26T04:17:51.329674+0000","last_became_active":"2024-03-26T04:17:11.138456+0000","last_became_peered":"2024-03-26T04:17:11.138456+0000","last_unstale":"2024-03-26T04:17:51.329674+0000","last_undegraded":"2024-03-26T04:17:51.329674+0000","last_fullsized":"2024-03-26T04:17:51.329674+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:35:22.875520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":50,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.238034+0000","last_change":"2024-03-26T04:17:13.153000+0000","last_active":"2024-03-26T04:17:51.238034+0000","last_peered":"2024-03-26T04:17:51.238034+0000","last_clean":"2024-03-26T04:17:51.238034+0000","last_became_active":"2024-03-26T04:17:13.152569+0000","last_became_peered":"2024-03-26T04:17:13.152569+0000","last_unstale":"2024-03-26T04:17:51.238034+0000","last_undegraded":"2024-03-26T04:17:51.238034+0000","last_fullsized":"2024-03-26T04:17:51.238034+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:08:43.535380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.330964+0000","last_change":"2024-03-26T04:17:17.256897+0000","last_active":"2024-03-26T04:17:51.330964+0000","last_peered":"2024-03-26T04:17:51.330964+0000","last_clean":"2024-03-26T04:17:51.330964+0000","last_became_active":"2024-03-26T04:17:17.256122+0000","last_became_peered":"2024-03-26T04:17:17.256122+0000","last_unstale":"2024-03-26T04:17:51.330964+0000","last_undegraded":"2024-03-26T04:17:51.330964+0000","last_fullsized":"2024-03-26T04:17:51.330964+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:19:22.207774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.032409+0000","last_change":"2024-03-26T04:17:15.168405+0000","last_active":"2024-03-26T04:17:51.032409+0000","last_peered":"2024-03-26T04:17:51.032409+0000","last_clean":"2024-03-26T04:17:51.032409+0000","last_became_active":"2024-03-26T04:17:15.168136+0000","last_became_peered":"2024-03-26T04:17:15.168136+0000","last_unstale":"2024-03-26T04:17:51.032409+0000","last_undegraded":"2024-03-26T04:17:51.032409+0000","last_fullsized":"2024-03-26T04:17:51.032409+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:43:05.542786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.145625+0000","last_change":"2024-03-26T04:17:11.507797+0000","last_active":"2024-03-26T04:17:51.145625+0000","last_peered":"2024-03-26T04:17:51.145625+0000","last_clean":"2024-03-26T04:17:51.145625+0000","last_became_active":"2024-03-26T04:17:11.507413+0000","last_became_peered":"2024-03-26T04:17:11.507413+0000","last_unstale":"2024-03-26T04:17:51.145625+0000","last_undegraded":"2024-03-26T04:17:51.145625+0000","last_fullsized":"2024-03-26T04:17:51.145625+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:32:21.104531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"68'17","reported_seq":56,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.146323+0000","last_change":"2024-03-26T04:17:13.144972+0000","last_active":"2024-03-26T04:17:51.146323+0000","last_peered":"2024-03-26T04:17:51.146323+0000","last_clean":"2024-03-26T04:17:51.146323+0000","last_became_active":"2024-03-26T04:17:13.144563+0000","last_became_peered":"2024-03-26T04:17:13.144563+0000","last_unstale":"2024-03-26T04:17:51.146323+0000","last_undegraded":"2024-03-26T04:17:51.146323+0000","last_fullsized":"2024-03-26T04:17:51.146323+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:51:42.089112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.040549+0000","last_change":"2024-03-26T04:17:17.176526+0000","last_active":"2024-03-26T04:17:51.040549+0000","last_peered":"2024-03-26T04:17:51.040549+0000","last_clean":"2024-03-26T04:17:51.040549+0000","last_became_active":"2024-03-26T04:17:17.174946+0000","last_became_peered":"2024-03-26T04:17:17.174946+0000","last_unstale":"2024-03-26T04:17:51.040549+0000","last_undegraded":"2024-03-26T04:17:51.040549+0000","last_fullsized":"2024-03-26T04:17:51.040549+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:33:54.106103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.241590+0000","last_change":"2024-03-26T04:17:15.171000+0000","last_active":"2024-03-26T04:17:51.241590+0000","last_peered":"2024-03-26T04:17:51.241590+0000","last_clean":"2024-03-26T04:17:51.241590+0000","last_became_active":"2024-03-26T04:17:15.170655+0000","last_became_peered":"2024-03-26T04:17:15.170655+0000","last_unstale":"2024-03-26T04:17:51.241590+0000","last_undegraded":"2024-03-26T04:17:51.241590+0000","last_fullsized":"2024-03-26T04:17:51.241590+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:26:50.974086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.330598+0000","last_change":"2024-03-26T04:17:11.140891+0000","last_active":"2024-03-26T04:17:51.330598+0000","last_peered":"2024-03-26T04:17:51.330598+0000","last_clean":"2024-03-26T04:17:51.330598+0000","last_became_active":"2024-03-26T04:17:11.140598+0000","last_became_peered":"2024-03-26T04:17:11.140598+0000","last_unstale":"2024-03-26T04:17:51.330598+0000","last_undegraded":"2024-03-26T04:17:51.330598+0000","last_fullsized":"2024-03-26T04:17:51.330598+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:05:09.457124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":53,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.146545+0000","last_change":"2024-03-26T04:17:13.146593+0000","last_active":"2024-03-26T04:17:51.146545+0000","last_peered":"2024-03-26T04:17:51.146545+0000","last_clean":"2024-03-26T04:17:51.146545+0000","last_became_active":"2024-03-26T04:17:13.146288+0000","last_became_peered":"2024-03-26T04:17:13.146288+0000","last_unstale":"2024-03-26T04:17:51.146545+0000","last_undegraded":"2024-03-26T04:17:51.146545+0000","last_fullsized":"2024-03-26T04:17:51.146545+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:53:16.752017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.146469+0000","last_change":"2024-03-26T04:17:17.175151+0000","last_active":"2024-03-26T04:17:51.146469+0000","last_peered":"2024-03-26T04:17:51.146469+0000","last_clean":"2024-03-26T04:17:51.146469+0000","last_became_active":"2024-03-26T04:17:17.174794+0000","last_became_peered":"2024-03-26T04:17:17.174794+0000","last_unstale":"2024-03-26T04:17:51.146469+0000","last_undegraded":"2024-03-26T04:17:51.146469+0000","last_fullsized":"2024-03-26T04:17:51.146469+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:15:45.052823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"68'8","reported_seq":45,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:18:21.559274+0000","last_change":"2024-03-26T04:17:15.170623+0000","last_active":"2024-03-26T04:18:21.559274+0000","last_peered":"2024-03-26T04:18:21.559274+0000","last_clean":"2024-03-26T04:18:21.559274+0000","last_became_active":"2024-03-26T04:17:15.170254+0000","last_became_peered":"2024-03-26T04:17:15.170254+0000","last_unstale":"2024-03-26T04:18:21.559274+0000","last_undegraded":"2024-03-26T04:18:21.559274+0000","last_fullsized":"2024-03-26T04:18:21.559274+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:29:10.632688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.329698+0000","last_change":"2024-03-26T04:17:11.140053+0000","last_active":"2024-03-26T04:17:51.329698+0000","last_peered":"2024-03-26T04:17:51.329698+0000","last_clean":"2024-03-26T04:17:51.329698+0000","last_became_active":"2024-03-26T04:17:11.139726+0000","last_became_peered":"2024-03-26T04:17:11.139726+0000","last_unstale":"2024-03-26T04:17:51.329698+0000","last_undegraded":"2024-03-26T04:17:51.329698+0000","last_fullsized":"2024-03-26T04:17:51.329698+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:51:35.051518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":40,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.146047+0000","last_change":"2024-03-26T04:17:13.150909+0000","last_active":"2024-03-26T04:17:51.146047+0000","last_peered":"2024-03-26T04:17:51.146047+0000","last_clean":"2024-03-26T04:17:51.146047+0000","last_became_active":"2024-03-26T04:17:13.150582+0000","last_became_peered":"2024-03-26T04:17:13.150582+0000","last_unstale":"2024-03-26T04:17:51.146047+0000","last_undegraded":"2024-03-26T04:17:51.146047+0000","last_fullsized":"2024-03-26T04:17:51.146047+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:26:13.619244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":587,"ondisk_log_size":587,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":64,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234004,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32432,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712720,"statfs":{"total":95995035648,"available":95961825280,"internally_reserved":0,"allocated":1286144,"data_stored":859743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495641,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32416,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712736,"statfs":{"total":95995035648,"available":95961841664,"internally_reserved":0,"allocated":1265664,"data_stored":858691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27748,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717404,"statfs":{"total":95995035648,"available":95966621696,"internally_reserved":0,"allocated":745472,"data_stored":270519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738422,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32440,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712712,"statfs":{"total":95995035648,"available":95961817088,"internally_reserved":0,"allocated":1294336,"data_stored":860360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411373,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27680,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717472,"statfs":{"total":95995035648,"available":95966691328,"internally_reserved":0,"allocated":675840,"data_stored":269711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182440,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31876,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713276,"statfs":{"total":95995035648,"available":95962394624,"internally_reserved":0,"allocated":712704,"data_stored":269227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018914,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31868,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713284,"statfs":{"total":95995035648,"available":95962402816,"internally_reserved":0,"allocated":704512,"data_stored":269326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757276,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27688,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717464,"statfs":{"total":95995035648,"available":95966683136,"internally_reserved":0,"allocated":684032,"data_stored":269537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T04:18:33.369 DEBUG:teuthology.orchestra.run.smithi016:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph pg dump --format=json 2024-03-26T04:18:33.587 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:33 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:18:33.587 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:33 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:18:33.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:33 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:18:34.475 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:18:34.603 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:34 smithi016 ceph-mon[37215]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:18:34.603 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:34 smithi016 ceph-mon[29101]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:18:34.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:34 smithi183 ceph-mon[33745]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:18:36.436 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:18:36 smithi016 podman[72083]: 2024-03-26 04:18:36.35730089 +0000 UTC m=+0.473153140 container create 7cd00f19dd8a9e5b92136fa7669422dd9c52af1d669b75c82e4495a73f9c033e (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:18:36.686 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:36 smithi016 ceph-mon[37215]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:18:36.686 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:36 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:18:36.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:36 smithi016 ceph-mon[29101]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:18:36.686 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:36 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:18:36.686 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:18:36 smithi016 podman[72083]: 2024-03-26 04:18:36.668719024 +0000 UTC m=+0.784571233 container init 7cd00f19dd8a9e5b92136fa7669422dd9c52af1d669b75c82e4495a73f9c033e (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:18:36.687 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:18:36 smithi016 podman[72083]: 2024-03-26 04:18:36.671629993 +0000 UTC m=+0.787482203 container start 7cd00f19dd8a9e5b92136fa7669422dd9c52af1d669b75c82e4495a73f9c033e (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:18:36.687 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:18:36 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-alertmanager-a[72105]: ts=2024-03-26T04:18:36.684Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-26T04:18:36.687 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:18:36 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-alertmanager-a[72105]: ts=2024-03-26T04:18:36.684Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-26T04:18:36.687 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:18:36 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-alertmanager-a[72105]: ts=2024-03-26T04:18:36.685Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.16 port=9094 2024-03-26T04:18:36.687 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:18:36 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-alertmanager-a[72105]: ts=2024-03-26T04:18:36.686Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-26T04:18:36.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:36 smithi183 ceph-mon[33745]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:18:36.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:36 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:18:36.942 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:18:36 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-alertmanager-a[72105]: ts=2024-03-26T04:18:36.710Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T04:18:36.943 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:18:36 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-alertmanager-a[72105]: ts=2024-03-26T04:18:36.710Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T04:18:36.943 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:18:36 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-alertmanager-a[72105]: ts=2024-03-26T04:18:36.713Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-26T04:18:36.943 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:18:36 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-alertmanager-a[72105]: ts=2024-03-26T04:18:36.713Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-26T04:18:36.943 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:18:36 smithi016 bash[72083]: 7cd00f19dd8a9e5b92136fa7669422dd9c52af1d669b75c82e4495a73f9c033e 2024-03-26T04:18:36.943 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:18:36 smithi016 systemd[1]: Started Ceph alertmanager.a for d7d59826-eb26-11ee-95d0-87774f69a715. 2024-03-26T04:18:37.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.028+0000 7fb6ab726700 1 -- 172.21.15.16:0/2981960841 >> v1:172.21.15.16:6790/0 conn(0x7fb6a410d7a0 legacy=0x7fb6a410fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:37.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.028+0000 7fb6ab726700 1 -- 172.21.15.16:0/2981960841 shutdown_connections 2024-03-26T04:18:37.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.028+0000 7fb6ab726700 1 -- 172.21.15.16:0/2981960841 >> 172.21.15.16:0/2981960841 conn(0x7fb6a40fc820 msgr2=0x7fb6a40fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:18:37.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.028+0000 7fb6ab726700 1 -- 172.21.15.16:0/2981960841 shutdown_connections 2024-03-26T04:18:37.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.028+0000 7fb6ab726700 1 -- 172.21.15.16:0/2981960841 wait complete. 2024-03-26T04:18:37.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.029+0000 7fb6ab726700 1 Processor -- start 2024-03-26T04:18:37.037 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.029+0000 7fb6ab726700 1 -- start start 2024-03-26T04:18:37.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.029+0000 7fb6ab726700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb6a41a6c40 con 0x7fb6a4100f70 2024-03-26T04:18:37.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.029+0000 7fb6ab726700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb6a41a7e20 con 0x7fb6a4109c00 2024-03-26T04:18:37.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.029+0000 7fb6ab726700 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fb6a41a9020 con 0x7fb6a410d7a0 2024-03-26T04:18:37.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.029+0000 7fb6a94c2700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb6a4100f70 0x7fb6a41061b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47910/0 (socket says 172.21.15.16:47910) 2024-03-26T04:18:37.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.029+0000 7fb6a94c2700 1 -- 172.21.15.16:0/6799838 learned_addr learned my addr 172.21.15.16:0/6799838 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:18:37.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.029+0000 7fb6967fc700 1 -- 172.21.15.16:0/6799838 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3419138832 0 0) 0x7fb6a41a6c40 con 0x7fb6a4100f70 2024-03-26T04:18:37.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.029+0000 7fb6967fc700 1 -- 172.21.15.16:0/6799838 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6900035b0 con 0x7fb6a4100f70 2024-03-26T04:18:37.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.029+0000 7fb6967fc700 1 -- 172.21.15.16:0/6799838 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3514150115 0 0) 0x7fb6a41a9020 con 0x7fb6a410d7a0 2024-03-26T04:18:37.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.029+0000 7fb6967fc700 1 -- 172.21.15.16:0/6799838 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6a41a6c40 con 0x7fb6a410d7a0 2024-03-26T04:18:37.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.029+0000 7fb6967fc700 1 -- 172.21.15.16:0/6799838 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1306033161 0 0) 0x7fb6a41a7e20 con 0x7fb6a4109c00 2024-03-26T04:18:37.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.029+0000 7fb6967fc700 1 -- 172.21.15.16:0/6799838 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6a41a9020 con 0x7fb6a4109c00 2024-03-26T04:18:37.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.029+0000 7fb6967fc700 1 -- 172.21.15.16:0/6799838 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1528374096 0 0) 0x7fb6900035b0 con 0x7fb6a4100f70 2024-03-26T04:18:37.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.029+0000 7fb6967fc700 1 -- 172.21.15.16:0/6799838 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6a41a7e20 con 0x7fb6a4100f70 2024-03-26T04:18:37.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.029+0000 7fb6967fc700 1 -- 172.21.15.16:0/6799838 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fb6a0001950 con 0x7fb6a4100f70 2024-03-26T04:18:37.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.030+0000 7fb6967fc700 1 -- 172.21.15.16:0/6799838 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1428710714 0 0) 0x7fb6a41a6c40 con 0x7fb6a410d7a0 2024-03-26T04:18:37.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.030+0000 7fb6967fc700 1 -- 172.21.15.16:0/6799838 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6900035b0 con 0x7fb6a410d7a0 2024-03-26T04:18:37.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.030+0000 7fb6967fc700 1 -- 172.21.15.16:0/6799838 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fb69c001890 con 0x7fb6a410d7a0 2024-03-26T04:18:37.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.030+0000 7fb6967fc700 1 -- 172.21.15.16:0/6799838 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4158265008 0 0) 0x7fb6a41a9020 con 0x7fb6a4109c00 2024-03-26T04:18:37.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.030+0000 7fb6967fc700 1 -- 172.21.15.16:0/6799838 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6a41a6c40 con 0x7fb6a4109c00 2024-03-26T04:18:37.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.030+0000 7fb6967fc700 1 -- 172.21.15.16:0/6799838 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fb6980017b0 con 0x7fb6a4109c00 2024-03-26T04:18:37.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.030+0000 7fb6967fc700 1 -- 172.21.15.16:0/6799838 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4144663608 0 0) 0x7fb6a41a7e20 con 0x7fb6a4100f70 2024-03-26T04:18:37.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.030+0000 7fb6967fc700 1 -- 172.21.15.16:0/6799838 >> v1:172.21.15.16:6790/0 conn(0x7fb6a410d7a0 legacy=0x7fb6a41a5300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:37.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.030+0000 7fb6967fc700 1 -- 172.21.15.16:0/6799838 >> v1:172.21.15.183:6789/0 conn(0x7fb6a4109c00 legacy=0x7fb6a41a1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:37.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.030+0000 7fb6967fc700 1 -- 172.21.15.16:0/6799838 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6a41aa220 con 0x7fb6a4100f70 2024-03-26T04:18:37.039 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.030+0000 7fb6ab726700 1 -- 172.21.15.16:0/6799838 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6a41a9210 con 0x7fb6a4100f70 2024-03-26T04:18:37.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.030+0000 7fb6ab726700 1 -- 172.21.15.16:0/6799838 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6a41a9690 con 0x7fb6a4100f70 2024-03-26T04:18:37.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.030+0000 7fb6967fc700 1 -- 172.21.15.16:0/6799838 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7fb6a0001c40 con 0x7fb6a4100f70 2024-03-26T04:18:37.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.030+0000 7fb6967fc700 1 -- 172.21.15.16:0/6799838 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fb6a0005c60 con 0x7fb6a4100f70 2024-03-26T04:18:37.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.031+0000 7fb6ab726700 1 -- 172.21.15.16:0/6799838 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb688005120 con 0x7fb6a4100f70 2024-03-26T04:18:37.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.031+0000 7fb6967fc700 1 -- 172.21.15.16:0/6799838 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 22) v1 ==== 91420+0+0 (unknown 3095259778 0 0) 0x7fb6a0003930 con 0x7fb6a4100f70 2024-03-26T04:18:37.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.032+0000 7fb6967fc700 1 -- 172.21.15.16:0/6799838 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(70..70 src has 1..70) v4 ==== 6172+0+0 (unknown 1260882646 0 0) 0x7fb6a00888f0 con 0x7fb6a4100f70 2024-03-26T04:18:37.040 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.036+0000 7fb6967fc700 1 -- 172.21.15.16:0/6799838 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fb6a00559a0 con 0x7fb6a4100f70 2024-03-26T04:18:37.237 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.236+0000 7fb6ab726700 1 -- 172.21.15.16:0/6799838 --> v1:172.21.15.16:6800/3868004154 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb688000b90 con 0x7fb69006eb10 2024-03-26T04:18:37.237 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:18:36 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: [26/Mar/2024:04:18:36] ENGINE Bus STOPPING 2024-03-26T04:18:37.237 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:18:36 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: [26/Mar/2024:04:18:36] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T04:18:37.237 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:18:36 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: [26/Mar/2024:04:18:36] ENGINE Bus STOPPED 2024-03-26T04:18:37.237 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:18:36 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: [26/Mar/2024:04:18:36] ENGINE Bus STARTING 2024-03-26T04:18:37.237 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:18:37 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: [26/Mar/2024:04:18:37] ENGINE Serving on http://:::9283 2024-03-26T04:18:37.237 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:18:37 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: [26/Mar/2024:04:18:37] ENGINE Bus STARTED 2024-03-26T04:18:37.246 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.246+0000 7fb6967fc700 1 -- 172.21.15.16:0/6799838 <== mgr.14616 v1:172.21.15.16:6800/3868004154 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+346507 (unknown 2965378022 0 697161303) 0x7fb688000b90 con 0x7fb69006eb10 2024-03-26T04:18:37.251 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:18:37.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.249+0000 7fb6ab726700 1 -- 172.21.15.16:0/6799838 >> v1:172.21.15.16:6800/3868004154 conn(0x7fb69006eb10 legacy=0x7fb690070fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:37.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.249+0000 7fb6ab726700 1 -- 172.21.15.16:0/6799838 >> v1:172.21.15.16:6789/0 conn(0x7fb6a4100f70 legacy=0x7fb6a41061b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:37.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.249+0000 7fb6ab726700 1 -- 172.21.15.16:0/6799838 shutdown_connections 2024-03-26T04:18:37.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.249+0000 7fb6ab726700 1 -- 172.21.15.16:0/6799838 >> 172.21.15.16:0/6799838 conn(0x7fb6a40fc820 msgr2=0x7fb6a40ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:18:37.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.250+0000 7fb6ab726700 1 -- 172.21.15.16:0/6799838 shutdown_connections 2024-03-26T04:18:37.254 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:37.250+0000 7fb6ab726700 1 -- 172.21.15.16:0/6799838 wait complete. 2024-03-26T04:18:37.254 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-03-26T04:18:38.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:37 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:38.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:37 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:38.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:37 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:38.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:37 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:38.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:37 smithi016 ceph-mon[37215]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T04:18:38.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:37 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:38.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:37 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:38.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:37 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T04:18:38.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:37 smithi016 ceph-mon[37215]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T04:18:38.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:37 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:38.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:37 smithi016 ceph-mon[37215]: Deploying daemon grafana.a on smithi183 2024-03-26T04:18:38.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:37 smithi016 ceph-mon[37215]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:18:38.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:37 smithi016 ceph-mon[37215]: from='client.14766 v1:172.21.15.16:0/6799838' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:18:38.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:37 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:38.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:37 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:38.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:37 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:38.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:37 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:38.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:37 smithi016 ceph-mon[29101]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T04:18:38.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:37 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:38.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:37 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:38.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:37 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T04:18:38.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:37 smithi016 ceph-mon[29101]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T04:18:38.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:37 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:38.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:37 smithi016 ceph-mon[29101]: Deploying daemon grafana.a on smithi183 2024-03-26T04:18:38.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:37 smithi016 ceph-mon[29101]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:18:38.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:37 smithi016 ceph-mon[29101]: from='client.14766 v1:172.21.15.16:0/6799838' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:18:38.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:37 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:38.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:37 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:38.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:37 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:38.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:37 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:38.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:37 smithi183 ceph-mon[33745]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-26T04:18:38.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:37 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:38.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:37 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:38.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:37 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T04:18:38.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:37 smithi183 ceph-mon[33745]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-26T04:18:38.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:37 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:38.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:37 smithi183 ceph-mon[33745]: Deploying daemon grafana.a on smithi183 2024-03-26T04:18:38.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:37 smithi183 ceph-mon[33745]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:18:38.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:37 smithi183 ceph-mon[33745]: from='client.14766 v1:172.21.15.16:0/6799838' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-26T04:18:38.319 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-03-26T04:18:36.953310+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596318,"num_objects":230,"num_object_clones":0,"num_object_copies":690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":230,"num_whiteouts":0,"num_read":1301,"num_read_kb":1115,"num_write":807,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":794,"ondisk_log_size":794,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":244148,"kb_used_data":7196,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":749717068,"statfs":{"total":767960285184,"available":767710277632,"internally_reserved":0,"allocated":7368704,"data_stored":3927114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":242535984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002992"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.145703+0000","last_change":"2024-03-26T04:17:17.170196+0000","last_active":"2024-03-26T04:17:51.145703+0000","last_peered":"2024-03-26T04:17:51.145703+0000","last_clean":"2024-03-26T04:17:51.145703+0000","last_became_active":"2024-03-26T04:17:17.169900+0000","last_became_peered":"2024-03-26T04:17:17.169900+0000","last_unstale":"2024-03-26T04:17:51.145703+0000","last_undegraded":"2024-03-26T04:17:51.145703+0000","last_fullsized":"2024-03-26T04:17:51.145703+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:50:37.434494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.239613+0000","last_change":"2024-03-26T04:17:15.171595+0000","last_active":"2024-03-26T04:17:51.239613+0000","last_peered":"2024-03-26T04:17:51.239613+0000","last_clean":"2024-03-26T04:17:51.239613+0000","last_became_active":"2024-03-26T04:17:15.171277+0000","last_became_peered":"2024-03-26T04:17:15.171277+0000","last_unstale":"2024-03-26T04:17:51.239613+0000","last_undegraded":"2024-03-26T04:17:51.239613+0000","last_fullsized":"2024-03-26T04:17:51.239613+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:28:30.163211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"61'1","reported_seq":35,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.929898+0000","last_change":"2024-03-26T04:17:11.507950+0000","last_active":"2024-03-26T04:17:50.929898+0000","last_peered":"2024-03-26T04:17:50.929898+0000","last_clean":"2024-03-26T04:17:50.929898+0000","last_became_active":"2024-03-26T04:17:11.507622+0000","last_became_peered":"2024-03-26T04:17:11.507622+0000","last_unstale":"2024-03-26T04:17:50.929898+0000","last_undegraded":"2024-03-26T04:17:50.929898+0000","last_fullsized":"2024-03-26T04:17:50.929898+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:50:35.698244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"68'15","reported_seq":53,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.145487+0000","last_change":"2024-03-26T04:17:13.145335+0000","last_active":"2024-03-26T04:17:51.145487+0000","last_peered":"2024-03-26T04:17:51.145487+0000","last_clean":"2024-03-26T04:17:51.145487+0000","last_became_active":"2024-03-26T04:17:13.144919+0000","last_became_peered":"2024-03-26T04:17:13.144919+0000","last_unstale":"2024-03-26T04:17:51.145487+0000","last_undegraded":"2024-03-26T04:17:51.145487+0000","last_fullsized":"2024-03-26T04:17:51.145487+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:23:37.226689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.330673+0000","last_change":"2024-03-26T04:17:17.258204+0000","last_active":"2024-03-26T04:17:51.330673+0000","last_peered":"2024-03-26T04:17:51.330673+0000","last_clean":"2024-03-26T04:17:51.330673+0000","last_became_active":"2024-03-26T04:17:17.257849+0000","last_became_peered":"2024-03-26T04:17:17.257849+0000","last_unstale":"2024-03-26T04:17:51.330673+0000","last_undegraded":"2024-03-26T04:17:51.330673+0000","last_fullsized":"2024-03-26T04:17:51.330673+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:32:38.220150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.145491+0000","last_change":"2024-03-26T04:17:15.169117+0000","last_active":"2024-03-26T04:17:51.145491+0000","last_peered":"2024-03-26T04:17:51.145491+0000","last_clean":"2024-03-26T04:17:51.145491+0000","last_became_active":"2024-03-26T04:17:15.168745+0000","last_became_peered":"2024-03-26T04:17:15.168745+0000","last_unstale":"2024-03-26T04:17:51.145491+0000","last_undegraded":"2024-03-26T04:17:51.145491+0000","last_fullsized":"2024-03-26T04:17:51.145491+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:30:25.756222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.331278+0000","last_change":"2024-03-26T04:17:11.507897+0000","last_active":"2024-03-26T04:17:51.331278+0000","last_peered":"2024-03-26T04:17:51.331278+0000","last_clean":"2024-03-26T04:17:51.331278+0000","last_became_active":"2024-03-26T04:17:11.507487+0000","last_became_peered":"2024-03-26T04:17:11.507487+0000","last_unstale":"2024-03-26T04:17:51.331278+0000","last_undegraded":"2024-03-26T04:17:51.331278+0000","last_fullsized":"2024-03-26T04:17:51.331278+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:16:56.597175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"68'39","reported_seq":98,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.330472+0000","last_change":"2024-03-26T04:17:13.154016+0000","last_active":"2024-03-26T04:17:51.330472+0000","last_peered":"2024-03-26T04:17:51.330472+0000","last_clean":"2024-03-26T04:17:51.330472+0000","last_became_active":"2024-03-26T04:17:13.153600+0000","last_became_peered":"2024-03-26T04:17:13.153600+0000","last_unstale":"2024-03-26T04:17:51.330472+0000","last_undegraded":"2024-03-26T04:17:51.330472+0000","last_fullsized":"2024-03-26T04:17:51.330472+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:30:30.806406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":62,"num_read_kb":49,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.928629+0000","last_change":"2024-03-26T04:17:17.175565+0000","last_active":"2024-03-26T04:17:50.928629+0000","last_peered":"2024-03-26T04:17:50.928629+0000","last_clean":"2024-03-26T04:17:50.928629+0000","last_became_active":"2024-03-26T04:17:17.175116+0000","last_became_peered":"2024-03-26T04:17:17.175116+0000","last_unstale":"2024-03-26T04:17:50.928629+0000","last_undegraded":"2024-03-26T04:17:50.928629+0000","last_fullsized":"2024-03-26T04:17:50.928629+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:42:41.024222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.031867+0000","last_change":"2024-03-26T04:17:15.166332+0000","last_active":"2024-03-26T04:17:51.031867+0000","last_peered":"2024-03-26T04:17:51.031867+0000","last_clean":"2024-03-26T04:17:51.031867+0000","last_became_active":"2024-03-26T04:17:15.166001+0000","last_became_peered":"2024-03-26T04:17:15.166001+0000","last_unstale":"2024-03-26T04:17:51.031867+0000","last_undegraded":"2024-03-26T04:17:51.031867+0000","last_fullsized":"2024-03-26T04:17:51.031867+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:51:07.301584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.032004+0000","last_change":"2024-03-26T04:17:11.136554+0000","last_active":"2024-03-26T04:17:51.032004+0000","last_peered":"2024-03-26T04:17:51.032004+0000","last_clean":"2024-03-26T04:17:51.032004+0000","last_became_active":"2024-03-26T04:17:11.136263+0000","last_became_peered":"2024-03-26T04:17:11.136263+0000","last_unstale":"2024-03-26T04:17:51.032004+0000","last_undegraded":"2024-03-26T04:17:51.032004+0000","last_fullsized":"2024-03-26T04:17:51.032004+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:58:56.300937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"68'21","reported_seq":65,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.146039+0000","last_change":"2024-03-26T04:17:13.150081+0000","last_active":"2024-03-26T04:17:51.146039+0000","last_peered":"2024-03-26T04:17:51.146039+0000","last_clean":"2024-03-26T04:17:51.146039+0000","last_became_active":"2024-03-26T04:17:13.149623+0000","last_became_peered":"2024-03-26T04:17:13.149623+0000","last_unstale":"2024-03-26T04:17:51.146039+0000","last_undegraded":"2024-03-26T04:17:51.146039+0000","last_fullsized":"2024-03-26T04:17:51.146039+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:12:11.987017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.031618+0000","last_change":"2024-03-26T04:17:17.168717+0000","last_active":"2024-03-26T04:17:51.031618+0000","last_peered":"2024-03-26T04:17:51.031618+0000","last_clean":"2024-03-26T04:17:51.031618+0000","last_became_active":"2024-03-26T04:17:17.168428+0000","last_became_peered":"2024-03-26T04:17:17.168428+0000","last_unstale":"2024-03-26T04:17:51.031618+0000","last_undegraded":"2024-03-26T04:17:51.031618+0000","last_fullsized":"2024-03-26T04:17:51.031618+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:46:43.689601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.931351+0000","last_change":"2024-03-26T04:17:15.172381+0000","last_active":"2024-03-26T04:17:50.931351+0000","last_peered":"2024-03-26T04:17:50.931351+0000","last_clean":"2024-03-26T04:17:50.931351+0000","last_became_active":"2024-03-26T04:17:15.172066+0000","last_became_peered":"2024-03-26T04:17:15.172066+0000","last_unstale":"2024-03-26T04:17:50.931351+0000","last_undegraded":"2024-03-26T04:17:50.931351+0000","last_fullsized":"2024-03-26T04:17:50.931351+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:56:02.948271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.031936+0000","last_change":"2024-03-26T04:17:11.136840+0000","last_active":"2024-03-26T04:17:51.031936+0000","last_peered":"2024-03-26T04:17:51.031936+0000","last_clean":"2024-03-26T04:17:51.031936+0000","last_became_active":"2024-03-26T04:17:11.135863+0000","last_became_peered":"2024-03-26T04:17:11.135863+0000","last_unstale":"2024-03-26T04:17:51.031936+0000","last_undegraded":"2024-03-26T04:17:51.031936+0000","last_fullsized":"2024-03-26T04:17:51.031936+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:43:04.439547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"68'11","reported_seq":47,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.145915+0000","last_change":"2024-03-26T04:17:13.150181+0000","last_active":"2024-03-26T04:17:51.145915+0000","last_peered":"2024-03-26T04:17:51.145915+0000","last_clean":"2024-03-26T04:17:51.145915+0000","last_became_active":"2024-03-26T04:17:13.149774+0000","last_became_peered":"2024-03-26T04:17:13.149774+0000","last_unstale":"2024-03-26T04:17:51.145915+0000","last_undegraded":"2024-03-26T04:17:51.145915+0000","last_fullsized":"2024-03-26T04:17:51.145915+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:08:47.713336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.146328+0000","last_change":"2024-03-26T04:17:17.256561+0000","last_active":"2024-03-26T04:17:51.146328+0000","last_peered":"2024-03-26T04:17:51.146328+0000","last_clean":"2024-03-26T04:17:51.146328+0000","last_became_active":"2024-03-26T04:17:17.256246+0000","last_became_peered":"2024-03-26T04:17:17.256246+0000","last_unstale":"2024-03-26T04:17:51.146328+0000","last_undegraded":"2024-03-26T04:17:51.146328+0000","last_fullsized":"2024-03-26T04:17:51.146328+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:41:04.652864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.237314+0000","last_change":"2024-03-26T04:17:15.172128+0000","last_active":"2024-03-26T04:17:51.237314+0000","last_peered":"2024-03-26T04:17:51.237314+0000","last_clean":"2024-03-26T04:17:51.237314+0000","last_became_active":"2024-03-26T04:17:15.170737+0000","last_became_peered":"2024-03-26T04:17:15.170737+0000","last_unstale":"2024-03-26T04:17:51.237314+0000","last_undegraded":"2024-03-26T04:17:51.237314+0000","last_fullsized":"2024-03-26T04:17:51.237314+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:06:42.242701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.929682+0000","last_change":"2024-03-26T04:17:11.137150+0000","last_active":"2024-03-26T04:17:50.929682+0000","last_peered":"2024-03-26T04:17:50.929682+0000","last_clean":"2024-03-26T04:17:50.929682+0000","last_became_active":"2024-03-26T04:17:11.136355+0000","last_became_peered":"2024-03-26T04:17:11.136355+0000","last_unstale":"2024-03-26T04:17:50.929682+0000","last_undegraded":"2024-03-26T04:17:50.929682+0000","last_fullsized":"2024-03-26T04:17:50.929682+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:33:13.638639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"68'15","reported_seq":50,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.040167+0000","last_change":"2024-03-26T04:17:13.150929+0000","last_active":"2024-03-26T04:17:51.040167+0000","last_peered":"2024-03-26T04:17:51.040167+0000","last_clean":"2024-03-26T04:17:51.040167+0000","last_became_active":"2024-03-26T04:17:13.150574+0000","last_became_peered":"2024-03-26T04:17:13.150574+0000","last_unstale":"2024-03-26T04:17:51.040167+0000","last_undegraded":"2024-03-26T04:17:51.040167+0000","last_fullsized":"2024-03-26T04:17:51.040167+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:11:06.417913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.331272+0000","last_change":"2024-03-26T04:17:17.256539+0000","last_active":"2024-03-26T04:17:51.331272+0000","last_peered":"2024-03-26T04:17:51.331272+0000","last_clean":"2024-03-26T04:17:51.331272+0000","last_became_active":"2024-03-26T04:17:17.256129+0000","last_became_peered":"2024-03-26T04:17:17.256129+0000","last_unstale":"2024-03-26T04:17:51.331272+0000","last_undegraded":"2024-03-26T04:17:51.331272+0000","last_fullsized":"2024-03-26T04:17:51.331272+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:12:09.148071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.145854+0000","last_change":"2024-03-26T04:17:15.169347+0000","last_active":"2024-03-26T04:17:51.145854+0000","last_peered":"2024-03-26T04:17:51.145854+0000","last_clean":"2024-03-26T04:17:51.145854+0000","last_became_active":"2024-03-26T04:17:15.169066+0000","last_became_peered":"2024-03-26T04:17:15.169066+0000","last_unstale":"2024-03-26T04:17:51.145854+0000","last_undegraded":"2024-03-26T04:17:51.145854+0000","last_fullsized":"2024-03-26T04:17:51.145854+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:52:49.722726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"61'1","reported_seq":35,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.146225+0000","last_change":"2024-03-26T04:17:11.507753+0000","last_active":"2024-03-26T04:17:51.146225+0000","last_peered":"2024-03-26T04:17:51.146225+0000","last_clean":"2024-03-26T04:17:51.146225+0000","last_became_active":"2024-03-26T04:17:11.507392+0000","last_became_peered":"2024-03-26T04:17:11.507392+0000","last_unstale":"2024-03-26T04:17:51.146225+0000","last_undegraded":"2024-03-26T04:17:51.146225+0000","last_fullsized":"2024-03-26T04:17:51.146225+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:54:24.277741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":398,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"68'18","reported_seq":60,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.330745+0000","last_change":"2024-03-26T04:17:13.153993+0000","last_active":"2024-03-26T04:17:51.330745+0000","last_peered":"2024-03-26T04:17:51.330745+0000","last_clean":"2024-03-26T04:17:51.330745+0000","last_became_active":"2024-03-26T04:17:13.153672+0000","last_became_peered":"2024-03-26T04:17:13.153672+0000","last_unstale":"2024-03-26T04:17:51.330745+0000","last_undegraded":"2024-03-26T04:17:51.330745+0000","last_fullsized":"2024-03-26T04:17:51.330745+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:14:55.758721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":34,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"68'1","reported_seq":18,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.931982+0000","last_change":"2024-03-26T04:17:17.175046+0000","last_active":"2024-03-26T04:17:50.931982+0000","last_peered":"2024-03-26T04:17:50.931982+0000","last_clean":"2024-03-26T04:17:50.931982+0000","last_became_active":"2024-03-26T04:17:17.174698+0000","last_became_peered":"2024-03-26T04:17:17.174698+0000","last_unstale":"2024-03-26T04:17:50.931982+0000","last_undegraded":"2024-03-26T04:17:50.931982+0000","last_fullsized":"2024-03-26T04:17:50.931982+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:04:46.045626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"68'8","reported_seq":50,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:18:31.561706+0000","last_change":"2024-03-26T04:17:15.172166+0000","last_active":"2024-03-26T04:18:31.561706+0000","last_peered":"2024-03-26T04:18:31.561706+0000","last_clean":"2024-03-26T04:18:31.561706+0000","last_became_active":"2024-03-26T04:17:15.171824+0000","last_became_peered":"2024-03-26T04:17:15.171824+0000","last_unstale":"2024-03-26T04:18:31.561706+0000","last_undegraded":"2024-03-26T04:18:31.561706+0000","last_fullsized":"2024-03-26T04:18:31.561706+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:33:31.669591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"61'1","reported_seq":30,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.239704+0000","last_change":"2024-03-26T04:17:11.140251+0000","last_active":"2024-03-26T04:17:51.239704+0000","last_peered":"2024-03-26T04:17:51.239704+0000","last_clean":"2024-03-26T04:17:51.239704+0000","last_became_active":"2024-03-26T04:17:11.139950+0000","last_became_peered":"2024-03-26T04:17:11.139950+0000","last_unstale":"2024-03-26T04:17:51.239704+0000","last_undegraded":"2024-03-26T04:17:51.239704+0000","last_fullsized":"2024-03-26T04:17:51.239704+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:46:15.986079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"68'16","reported_seq":52,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.930361+0000","last_change":"2024-03-26T04:17:13.893519+0000","last_active":"2024-03-26T04:17:50.930361+0000","last_peered":"2024-03-26T04:17:50.930361+0000","last_clean":"2024-03-26T04:17:50.930361+0000","last_became_active":"2024-03-26T04:17:13.893215+0000","last_became_peered":"2024-03-26T04:17:13.893215+0000","last_unstale":"2024-03-26T04:17:50.930361+0000","last_undegraded":"2024-03-26T04:17:50.930361+0000","last_fullsized":"2024-03-26T04:17:50.930361+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:53:01.567614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.237489+0000","last_change":"2024-03-26T04:17:17.168789+0000","last_active":"2024-03-26T04:17:51.237489+0000","last_peered":"2024-03-26T04:17:51.237489+0000","last_clean":"2024-03-26T04:17:51.237489+0000","last_became_active":"2024-03-26T04:17:17.168392+0000","last_became_peered":"2024-03-26T04:17:17.168392+0000","last_unstale":"2024-03-26T04:17:51.237489+0000","last_undegraded":"2024-03-26T04:17:51.237489+0000","last_fullsized":"2024-03-26T04:17:51.237489+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:37:24.318835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.930556+0000","last_change":"2024-03-26T04:17:15.167414+0000","last_active":"2024-03-26T04:17:50.930556+0000","last_peered":"2024-03-26T04:17:50.930556+0000","last_clean":"2024-03-26T04:17:50.930556+0000","last_became_active":"2024-03-26T04:17:15.167090+0000","last_became_peered":"2024-03-26T04:17:15.167090+0000","last_unstale":"2024-03-26T04:17:50.930556+0000","last_undegraded":"2024-03-26T04:17:50.930556+0000","last_fullsized":"2024-03-26T04:17:50.930556+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:04:04.380465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.329683+0000","last_change":"2024-03-26T04:17:11.136061+0000","last_active":"2024-03-26T04:17:51.329683+0000","last_peered":"2024-03-26T04:17:51.329683+0000","last_clean":"2024-03-26T04:17:51.329683+0000","last_became_active":"2024-03-26T04:17:11.135732+0000","last_became_peered":"2024-03-26T04:17:11.135732+0000","last_unstale":"2024-03-26T04:17:51.329683+0000","last_undegraded":"2024-03-26T04:17:51.329683+0000","last_fullsized":"2024-03-26T04:17:51.329683+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:48:33.762367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"68'19","reported_seq":58,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.331088+0000","last_change":"2024-03-26T04:17:13.147788+0000","last_active":"2024-03-26T04:17:51.331088+0000","last_peered":"2024-03-26T04:17:51.331088+0000","last_clean":"2024-03-26T04:17:51.331088+0000","last_became_active":"2024-03-26T04:17:13.147466+0000","last_became_peered":"2024-03-26T04:17:13.147466+0000","last_unstale":"2024-03-26T04:17:51.331088+0000","last_undegraded":"2024-03-26T04:17:51.331088+0000","last_fullsized":"2024-03-26T04:17:51.331088+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:47:37.680465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.931306+0000","last_change":"2024-03-26T04:17:17.174786+0000","last_active":"2024-03-26T04:17:50.931306+0000","last_peered":"2024-03-26T04:17:50.931306+0000","last_clean":"2024-03-26T04:17:50.931306+0000","last_became_active":"2024-03-26T04:17:17.174464+0000","last_became_peered":"2024-03-26T04:17:17.174464+0000","last_unstale":"2024-03-26T04:17:50.931306+0000","last_undegraded":"2024-03-26T04:17:50.931306+0000","last_fullsized":"2024-03-26T04:17:50.931306+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:40:29.421467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.330498+0000","last_change":"2024-03-26T04:17:15.171962+0000","last_active":"2024-03-26T04:17:51.330498+0000","last_peered":"2024-03-26T04:17:51.330498+0000","last_clean":"2024-03-26T04:17:51.330498+0000","last_became_active":"2024-03-26T04:17:15.171669+0000","last_became_peered":"2024-03-26T04:17:15.171669+0000","last_unstale":"2024-03-26T04:17:51.330498+0000","last_undegraded":"2024-03-26T04:17:51.330498+0000","last_fullsized":"2024-03-26T04:17:51.330498+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:33:59.964435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.929796+0000","last_change":"2024-03-26T04:17:11.136085+0000","last_active":"2024-03-26T04:17:50.929796+0000","last_peered":"2024-03-26T04:17:50.929796+0000","last_clean":"2024-03-26T04:17:50.929796+0000","last_became_active":"2024-03-26T04:17:11.135750+0000","last_became_peered":"2024-03-26T04:17:11.135750+0000","last_unstale":"2024-03-26T04:17:50.929796+0000","last_undegraded":"2024-03-26T04:17:50.929796+0000","last_fullsized":"2024-03-26T04:17:50.929796+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:52:57.756684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"68'4","reported_seq":31,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.329906+0000","last_change":"2024-03-26T04:17:13.155098+0000","last_active":"2024-03-26T04:17:51.329906+0000","last_peered":"2024-03-26T04:17:51.329906+0000","last_clean":"2024-03-26T04:17:51.329906+0000","last_became_active":"2024-03-26T04:17:13.154586+0000","last_became_peered":"2024-03-26T04:17:13.154586+0000","last_unstale":"2024-03-26T04:17:51.329906+0000","last_undegraded":"2024-03-26T04:17:51.329906+0000","last_fullsized":"2024-03-26T04:17:51.329906+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:09:09.517693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.330649+0000","last_change":"2024-03-26T04:17:17.256626+0000","last_active":"2024-03-26T04:17:51.330649+0000","last_peered":"2024-03-26T04:17:51.330649+0000","last_clean":"2024-03-26T04:17:51.330649+0000","last_became_active":"2024-03-26T04:17:17.256180+0000","last_became_peered":"2024-03-26T04:17:17.256180+0000","last_unstale":"2024-03-26T04:17:51.330649+0000","last_undegraded":"2024-03-26T04:17:51.330649+0000","last_fullsized":"2024-03-26T04:17:51.330649+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:14:04.504206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.931575+0000","last_change":"2024-03-26T04:17:15.174230+0000","last_active":"2024-03-26T04:17:50.931575+0000","last_peered":"2024-03-26T04:17:50.931575+0000","last_clean":"2024-03-26T04:17:50.931575+0000","last_became_active":"2024-03-26T04:17:15.173930+0000","last_became_peered":"2024-03-26T04:17:15.173930+0000","last_unstale":"2024-03-26T04:17:50.931575+0000","last_undegraded":"2024-03-26T04:17:50.931575+0000","last_fullsized":"2024-03-26T04:17:50.931575+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:51:42.310601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.032221+0000","last_change":"2024-03-26T04:17:11.138254+0000","last_active":"2024-03-26T04:17:51.032221+0000","last_peered":"2024-03-26T04:17:51.032221+0000","last_clean":"2024-03-26T04:17:51.032221+0000","last_became_active":"2024-03-26T04:17:11.137908+0000","last_became_peered":"2024-03-26T04:17:11.137908+0000","last_unstale":"2024-03-26T04:17:51.032221+0000","last_undegraded":"2024-03-26T04:17:51.032221+0000","last_fullsized":"2024-03-26T04:17:51.032221+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:25:22.795868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"68'17","reported_seq":56,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.330368+0000","last_change":"2024-03-26T04:17:13.894243+0000","last_active":"2024-03-26T04:17:51.330368+0000","last_peered":"2024-03-26T04:17:51.330368+0000","last_clean":"2024-03-26T04:17:51.330368+0000","last_became_active":"2024-03-26T04:17:13.893923+0000","last_became_peered":"2024-03-26T04:17:13.893923+0000","last_unstale":"2024-03-26T04:17:51.330368+0000","last_undegraded":"2024-03-26T04:17:51.330368+0000","last_fullsized":"2024-03-26T04:17:51.330368+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:48:34.406148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.928367+0000","last_change":"2024-03-26T04:17:17.176148+0000","last_active":"2024-03-26T04:17:50.928367+0000","last_peered":"2024-03-26T04:17:50.928367+0000","last_clean":"2024-03-26T04:17:50.928367+0000","last_became_active":"2024-03-26T04:17:17.175759+0000","last_became_peered":"2024-03-26T04:17:17.175759+0000","last_unstale":"2024-03-26T04:17:50.928367+0000","last_undegraded":"2024-03-26T04:17:50.928367+0000","last_fullsized":"2024-03-26T04:17:50.928367+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:01:17.171602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.330781+0000","last_change":"2024-03-26T04:17:15.174947+0000","last_active":"2024-03-26T04:17:51.330781+0000","last_peered":"2024-03-26T04:17:51.330781+0000","last_clean":"2024-03-26T04:17:51.330781+0000","last_became_active":"2024-03-26T04:17:15.174534+0000","last_became_peered":"2024-03-26T04:17:15.174534+0000","last_unstale":"2024-03-26T04:17:51.330781+0000","last_undegraded":"2024-03-26T04:17:51.330781+0000","last_fullsized":"2024-03-26T04:17:51.330781+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:32:57.104089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.932678+0000","last_change":"2024-03-26T04:17:11.135999+0000","last_active":"2024-03-26T04:17:50.932678+0000","last_peered":"2024-03-26T04:17:50.932678+0000","last_clean":"2024-03-26T04:17:50.932678+0000","last_became_active":"2024-03-26T04:17:11.135609+0000","last_became_peered":"2024-03-26T04:17:11.135609+0000","last_unstale":"2024-03-26T04:17:50.932678+0000","last_undegraded":"2024-03-26T04:17:50.932678+0000","last_fullsized":"2024-03-26T04:17:50.932678+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:32:12.218747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"68'9","reported_seq":41,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.240113+0000","last_change":"2024-03-26T04:17:13.144493+0000","last_active":"2024-03-26T04:17:51.240113+0000","last_peered":"2024-03-26T04:17:51.240113+0000","last_clean":"2024-03-26T04:17:51.240113+0000","last_became_active":"2024-03-26T04:17:13.144199+0000","last_became_peered":"2024-03-26T04:17:13.144199+0000","last_unstale":"2024-03-26T04:17:51.240113+0000","last_undegraded":"2024-03-26T04:17:51.240113+0000","last_fullsized":"2024-03-26T04:17:51.240113+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:22:58.859187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.330146+0000","last_change":"2024-03-26T04:17:17.181095+0000","last_active":"2024-03-26T04:17:51.330146+0000","last_peered":"2024-03-26T04:17:51.330146+0000","last_clean":"2024-03-26T04:17:51.330146+0000","last_became_active":"2024-03-26T04:17:17.180753+0000","last_became_peered":"2024-03-26T04:17:17.180753+0000","last_unstale":"2024-03-26T04:17:51.330146+0000","last_undegraded":"2024-03-26T04:17:51.330146+0000","last_fullsized":"2024-03-26T04:17:51.330146+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:34:56.012183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.239899+0000","last_change":"2024-03-26T04:17:15.173684+0000","last_active":"2024-03-26T04:17:51.239899+0000","last_peered":"2024-03-26T04:17:51.239899+0000","last_clean":"2024-03-26T04:17:51.239899+0000","last_became_active":"2024-03-26T04:17:15.173289+0000","last_became_peered":"2024-03-26T04:17:15.173289+0000","last_unstale":"2024-03-26T04:17:51.239899+0000","last_undegraded":"2024-03-26T04:17:51.239899+0000","last_fullsized":"2024-03-26T04:17:51.239899+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:35:13.135620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.145232+0000","last_change":"2024-03-26T04:17:11.140336+0000","last_active":"2024-03-26T04:17:51.145232+0000","last_peered":"2024-03-26T04:17:51.145232+0000","last_clean":"2024-03-26T04:17:51.145232+0000","last_became_active":"2024-03-26T04:17:11.139978+0000","last_became_peered":"2024-03-26T04:17:11.139978+0000","last_unstale":"2024-03-26T04:17:51.145232+0000","last_undegraded":"2024-03-26T04:17:51.145232+0000","last_fullsized":"2024-03-26T04:17:51.145232+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:14:43.523655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"68'11","reported_seq":44,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.146199+0000","last_change":"2024-03-26T04:17:13.144107+0000","last_active":"2024-03-26T04:17:51.146199+0000","last_peered":"2024-03-26T04:17:51.146199+0000","last_clean":"2024-03-26T04:17:51.146199+0000","last_became_active":"2024-03-26T04:17:13.143748+0000","last_became_peered":"2024-03-26T04:17:13.143748+0000","last_unstale":"2024-03-26T04:17:51.146199+0000","last_undegraded":"2024-03-26T04:17:51.146199+0000","last_fullsized":"2024-03-26T04:17:51.146199+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:58:01.925951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.930025+0000","last_change":"2024-03-26T04:17:17.179372+0000","last_active":"2024-03-26T04:17:50.930025+0000","last_peered":"2024-03-26T04:17:50.930025+0000","last_clean":"2024-03-26T04:17:50.930025+0000","last_became_active":"2024-03-26T04:17:17.179051+0000","last_became_peered":"2024-03-26T04:17:17.179051+0000","last_unstale":"2024-03-26T04:17:50.930025+0000","last_undegraded":"2024-03-26T04:17:50.930025+0000","last_fullsized":"2024-03-26T04:17:50.930025+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:46:32.062312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"68'8","reported_seq":47,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:18:31.561738+0000","last_change":"2024-03-26T04:17:15.169061+0000","last_active":"2024-03-26T04:18:31.561738+0000","last_peered":"2024-03-26T04:18:31.561738+0000","last_clean":"2024-03-26T04:18:31.561738+0000","last_became_active":"2024-03-26T04:17:15.168757+0000","last_became_peered":"2024-03-26T04:17:15.168757+0000","last_unstale":"2024-03-26T04:18:31.561738+0000","last_undegraded":"2024-03-26T04:18:31.561738+0000","last_fullsized":"2024-03-26T04:18:31.561738+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:31:54.756455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.931823+0000","last_change":"2024-03-26T04:17:11.510012+0000","last_active":"2024-03-26T04:17:50.931823+0000","last_peered":"2024-03-26T04:17:50.931823+0000","last_clean":"2024-03-26T04:17:50.931823+0000","last_became_active":"2024-03-26T04:17:11.509770+0000","last_became_peered":"2024-03-26T04:17:11.509770+0000","last_unstale":"2024-03-26T04:17:50.931823+0000","last_undegraded":"2024-03-26T04:17:50.931823+0000","last_fullsized":"2024-03-26T04:17:50.931823+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:23:14.674792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"68'22","reported_seq":64,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.330668+0000","last_change":"2024-03-26T04:17:13.892377+0000","last_active":"2024-03-26T04:17:51.330668+0000","last_peered":"2024-03-26T04:17:51.330668+0000","last_clean":"2024-03-26T04:17:51.330668+0000","last_became_active":"2024-03-26T04:17:13.892011+0000","last_became_peered":"2024-03-26T04:17:13.892011+0000","last_unstale":"2024-03-26T04:17:51.330668+0000","last_undegraded":"2024-03-26T04:17:51.330668+0000","last_fullsized":"2024-03-26T04:17:51.330668+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:46:59.513429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":33,"num_read_kb":26,"num_write":22,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.146560+0000","last_change":"2024-03-26T04:17:17.175008+0000","last_active":"2024-03-26T04:17:51.146560+0000","last_peered":"2024-03-26T04:17:51.146560+0000","last_clean":"2024-03-26T04:17:51.146560+0000","last_became_active":"2024-03-26T04:17:17.174716+0000","last_became_peered":"2024-03-26T04:17:17.174716+0000","last_unstale":"2024-03-26T04:17:51.146560+0000","last_undegraded":"2024-03-26T04:17:51.146560+0000","last_fullsized":"2024-03-26T04:17:51.146560+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:12:14.215487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"68'8","reported_seq":47,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:18:31.561520+0000","last_change":"2024-03-26T04:17:15.167221+0000","last_active":"2024-03-26T04:18:31.561520+0000","last_peered":"2024-03-26T04:18:31.561520+0000","last_clean":"2024-03-26T04:18:31.561520+0000","last_became_active":"2024-03-26T04:17:15.166917+0000","last_became_peered":"2024-03-26T04:17:15.166917+0000","last_unstale":"2024-03-26T04:18:31.561520+0000","last_undegraded":"2024-03-26T04:18:31.561520+0000","last_fullsized":"2024-03-26T04:18:31.561520+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:06:26.208161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.929103+0000","last_change":"2024-03-26T04:17:11.137236+0000","last_active":"2024-03-26T04:17:50.929103+0000","last_peered":"2024-03-26T04:17:50.929103+0000","last_clean":"2024-03-26T04:17:50.929103+0000","last_became_active":"2024-03-26T04:17:11.136939+0000","last_became_peered":"2024-03-26T04:17:11.136939+0000","last_unstale":"2024-03-26T04:17:50.929103+0000","last_undegraded":"2024-03-26T04:17:50.929103+0000","last_fullsized":"2024-03-26T04:17:50.929103+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:25:31.760614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"68'15","reported_seq":53,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.032349+0000","last_change":"2024-03-26T04:17:13.894345+0000","last_active":"2024-03-26T04:17:51.032349+0000","last_peered":"2024-03-26T04:17:51.032349+0000","last_clean":"2024-03-26T04:17:51.032349+0000","last_became_active":"2024-03-26T04:17:13.893928+0000","last_became_peered":"2024-03-26T04:17:13.893928+0000","last_unstale":"2024-03-26T04:17:51.032349+0000","last_undegraded":"2024-03-26T04:17:51.032349+0000","last_fullsized":"2024-03-26T04:17:51.032349+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:51:40.624569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.040322+0000","last_change":"2024-03-26T04:17:17.174273+0000","last_active":"2024-03-26T04:17:51.040322+0000","last_peered":"2024-03-26T04:17:51.040322+0000","last_clean":"2024-03-26T04:17:51.040322+0000","last_became_active":"2024-03-26T04:17:17.173960+0000","last_became_peered":"2024-03-26T04:17:17.173960+0000","last_unstale":"2024-03-26T04:17:51.040322+0000","last_undegraded":"2024-03-26T04:17:51.040322+0000","last_fullsized":"2024-03-26T04:17:51.040322+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:41:27.899074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.331533+0000","last_change":"2024-03-26T04:17:15.166097+0000","last_active":"2024-03-26T04:17:51.331533+0000","last_peered":"2024-03-26T04:17:51.331533+0000","last_clean":"2024-03-26T04:17:51.331533+0000","last_became_active":"2024-03-26T04:17:15.165753+0000","last_became_peered":"2024-03-26T04:17:15.165753+0000","last_unstale":"2024-03-26T04:17:51.331533+0000","last_undegraded":"2024-03-26T04:17:51.331533+0000","last_fullsized":"2024-03-26T04:17:51.331533+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:06:35.354370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.931373+0000","last_change":"2024-03-26T04:17:11.140113+0000","last_active":"2024-03-26T04:17:50.931373+0000","last_peered":"2024-03-26T04:17:50.931373+0000","last_clean":"2024-03-26T04:17:50.931373+0000","last_became_active":"2024-03-26T04:17:11.139843+0000","last_became_peered":"2024-03-26T04:17:11.139843+0000","last_unstale":"2024-03-26T04:17:50.931373+0000","last_undegraded":"2024-03-26T04:17:50.931373+0000","last_fullsized":"2024-03-26T04:17:50.931373+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:45:17.483706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"68'21","reported_seq":65,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.930435+0000","last_change":"2024-03-26T04:17:13.893466+0000","last_active":"2024-03-26T04:17:50.930435+0000","last_peered":"2024-03-26T04:17:50.930435+0000","last_clean":"2024-03-26T04:17:50.930435+0000","last_became_active":"2024-03-26T04:17:13.893181+0000","last_became_peered":"2024-03-26T04:17:13.893181+0000","last_unstale":"2024-03-26T04:17:50.930435+0000","last_undegraded":"2024-03-26T04:17:50.930435+0000","last_fullsized":"2024-03-26T04:17:50.930435+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:28:35.807341+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.932585+0000","last_change":"2024-03-26T04:17:17.258601+0000","last_active":"2024-03-26T04:17:50.932585+0000","last_peered":"2024-03-26T04:17:50.932585+0000","last_clean":"2024-03-26T04:17:50.932585+0000","last_became_active":"2024-03-26T04:17:17.258289+0000","last_became_peered":"2024-03-26T04:17:17.258289+0000","last_unstale":"2024-03-26T04:17:50.932585+0000","last_undegraded":"2024-03-26T04:17:50.932585+0000","last_fullsized":"2024-03-26T04:17:50.932585+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:25:22.120180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.032207+0000","last_change":"2024-03-26T04:17:15.170597+0000","last_active":"2024-03-26T04:17:51.032207+0000","last_peered":"2024-03-26T04:17:51.032207+0000","last_clean":"2024-03-26T04:17:51.032207+0000","last_became_active":"2024-03-26T04:17:15.170275+0000","last_became_peered":"2024-03-26T04:17:15.170275+0000","last_unstale":"2024-03-26T04:17:51.032207+0000","last_undegraded":"2024-03-26T04:17:51.032207+0000","last_fullsized":"2024-03-26T04:17:51.032207+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:24:38.102601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.330046+0000","last_change":"2024-03-26T04:17:11.138327+0000","last_active":"2024-03-26T04:17:51.330046+0000","last_peered":"2024-03-26T04:17:51.330046+0000","last_clean":"2024-03-26T04:17:51.330046+0000","last_became_active":"2024-03-26T04:17:11.138023+0000","last_became_peered":"2024-03-26T04:17:11.138023+0000","last_unstale":"2024-03-26T04:17:51.330046+0000","last_undegraded":"2024-03-26T04:17:51.330046+0000","last_fullsized":"2024-03-26T04:17:51.330046+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:48:15.477574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"68'12","reported_seq":46,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.331454+0000","last_change":"2024-03-26T04:17:13.892376+0000","last_active":"2024-03-26T04:17:51.331454+0000","last_peered":"2024-03-26T04:17:51.331454+0000","last_clean":"2024-03-26T04:17:51.331454+0000","last_became_active":"2024-03-26T04:17:13.892026+0000","last_became_peered":"2024-03-26T04:17:13.892026+0000","last_unstale":"2024-03-26T04:17:51.331454+0000","last_undegraded":"2024-03-26T04:17:51.331454+0000","last_fullsized":"2024-03-26T04:17:51.331454+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:45:38.509390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.031973+0000","last_change":"2024-03-26T04:17:17.256646+0000","last_active":"2024-03-26T04:17:51.031973+0000","last_peered":"2024-03-26T04:17:51.031973+0000","last_clean":"2024-03-26T04:17:51.031973+0000","last_became_active":"2024-03-26T04:17:17.256134+0000","last_became_peered":"2024-03-26T04:17:17.256134+0000","last_unstale":"2024-03-26T04:17:51.031973+0000","last_undegraded":"2024-03-26T04:17:51.031973+0000","last_fullsized":"2024-03-26T04:17:51.031973+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:38:49.190015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"61'2","reported_seq":41,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.932450+0000","last_change":"2024-03-26T04:17:11.142324+0000","last_active":"2024-03-26T04:17:50.932450+0000","last_peered":"2024-03-26T04:17:50.932450+0000","last_clean":"2024-03-26T04:17:50.932450+0000","last_became_active":"2024-03-26T04:17:11.142005+0000","last_became_peered":"2024-03-26T04:17:11.142005+0000","last_unstale":"2024-03-26T04:17:50.932450+0000","last_undegraded":"2024-03-26T04:17:50.932450+0000","last_fullsized":"2024-03-26T04:17:50.932450+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:43:35.639879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"68'27","reported_seq":74,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.031638+0000","last_change":"2024-03-26T04:17:13.894492+0000","last_active":"2024-03-26T04:17:51.031638+0000","last_peered":"2024-03-26T04:17:51.031638+0000","last_clean":"2024-03-26T04:17:51.031638+0000","last_became_active":"2024-03-26T04:17:13.893841+0000","last_became_peered":"2024-03-26T04:17:13.893841+0000","last_unstale":"2024-03-26T04:17:51.031638+0000","last_undegraded":"2024-03-26T04:17:51.031638+0000","last_fullsized":"2024-03-26T04:17:51.031638+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:40:39.438254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"68'8","reported_seq":50,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:18:31.561731+0000","last_change":"2024-03-26T04:17:15.172117+0000","last_active":"2024-03-26T04:18:31.561731+0000","last_peered":"2024-03-26T04:18:31.561731+0000","last_clean":"2024-03-26T04:18:31.561731+0000","last_became_active":"2024-03-26T04:17:15.171730+0000","last_became_peered":"2024-03-26T04:17:15.171730+0000","last_unstale":"2024-03-26T04:18:31.561731+0000","last_undegraded":"2024-03-26T04:18:31.561731+0000","last_fullsized":"2024-03-26T04:18:31.561731+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:26:00.568738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.330568+0000","last_change":"2024-03-26T04:17:17.177097+0000","last_active":"2024-03-26T04:17:51.330568+0000","last_peered":"2024-03-26T04:17:51.330568+0000","last_clean":"2024-03-26T04:17:51.330568+0000","last_became_active":"2024-03-26T04:17:17.176333+0000","last_became_peered":"2024-03-26T04:17:17.176333+0000","last_unstale":"2024-03-26T04:17:51.330568+0000","last_undegraded":"2024-03-26T04:17:51.330568+0000","last_fullsized":"2024-03-26T04:17:51.330568+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:53:08.534283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.931922+0000","last_change":"2024-03-26T04:17:11.143613+0000","last_active":"2024-03-26T04:17:50.931922+0000","last_peered":"2024-03-26T04:17:50.931922+0000","last_clean":"2024-03-26T04:17:50.931922+0000","last_became_active":"2024-03-26T04:17:11.143279+0000","last_became_peered":"2024-03-26T04:17:11.143279+0000","last_unstale":"2024-03-26T04:17:50.931922+0000","last_undegraded":"2024-03-26T04:17:50.931922+0000","last_fullsized":"2024-03-26T04:17:50.931922+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:23:16.431330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"68'24","reported_seq":72,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.145861+0000","last_change":"2024-03-26T04:17:13.150344+0000","last_active":"2024-03-26T04:17:51.145861+0000","last_peered":"2024-03-26T04:17:51.145861+0000","last_clean":"2024-03-26T04:17:51.145861+0000","last_became_active":"2024-03-26T04:17:13.149870+0000","last_became_peered":"2024-03-26T04:17:13.149870+0000","last_unstale":"2024-03-26T04:17:51.145861+0000","last_undegraded":"2024-03-26T04:17:51.145861+0000","last_fullsized":"2024-03-26T04:17:51.145861+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:19:52.903517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":32,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.039732+0000","last_change":"2024-03-26T04:17:15.165655+0000","last_active":"2024-03-26T04:17:51.039732+0000","last_peered":"2024-03-26T04:17:51.039732+0000","last_clean":"2024-03-26T04:17:51.039732+0000","last_became_active":"2024-03-26T04:17:15.165308+0000","last_became_peered":"2024-03-26T04:17:15.165308+0000","last_unstale":"2024-03-26T04:17:51.039732+0000","last_undegraded":"2024-03-26T04:17:51.039732+0000","last_fullsized":"2024-03-26T04:17:51.039732+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:42:38.061055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.931167+0000","last_change":"2024-03-26T04:17:17.176227+0000","last_active":"2024-03-26T04:17:50.931167+0000","last_peered":"2024-03-26T04:17:50.931167+0000","last_clean":"2024-03-26T04:17:50.931167+0000","last_became_active":"2024-03-26T04:17:17.175814+0000","last_became_peered":"2024-03-26T04:17:17.175814+0000","last_unstale":"2024-03-26T04:17:50.931167+0000","last_undegraded":"2024-03-26T04:17:50.931167+0000","last_fullsized":"2024-03-26T04:17:50.931167+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:22:47.174732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.040082+0000","last_change":"2024-03-26T04:17:15.170181+0000","last_active":"2024-03-26T04:17:51.040082+0000","last_peered":"2024-03-26T04:17:51.040082+0000","last_clean":"2024-03-26T04:17:51.040082+0000","last_became_active":"2024-03-26T04:17:15.169851+0000","last_became_peered":"2024-03-26T04:17:15.169851+0000","last_unstale":"2024-03-26T04:17:51.040082+0000","last_undegraded":"2024-03-26T04:17:51.040082+0000","last_fullsized":"2024-03-26T04:17:51.040082+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:47:48.162297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.932892+0000","last_change":"2024-03-26T04:17:11.140098+0000","last_active":"2024-03-26T04:17:50.932892+0000","last_peered":"2024-03-26T04:17:50.932892+0000","last_clean":"2024-03-26T04:17:50.932892+0000","last_became_active":"2024-03-26T04:17:11.139402+0000","last_became_peered":"2024-03-26T04:17:11.139402+0000","last_unstale":"2024-03-26T04:17:50.932892+0000","last_undegraded":"2024-03-26T04:17:50.932892+0000","last_fullsized":"2024-03-26T04:17:50.932892+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:38:39.696009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"68'25","reported_seq":68,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.330718+0000","last_change":"2024-03-26T04:17:13.892572+0000","last_active":"2024-03-26T04:17:51.330718+0000","last_peered":"2024-03-26T04:17:51.330718+0000","last_clean":"2024-03-26T04:17:51.330718+0000","last_became_active":"2024-03-26T04:17:13.892212+0000","last_became_peered":"2024-03-26T04:17:13.892212+0000","last_unstale":"2024-03-26T04:17:51.330718+0000","last_undegraded":"2024-03-26T04:17:51.330718+0000","last_fullsized":"2024-03-26T04:17:51.330718+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:50:49.538879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":41,"num_read_kb":29,"num_write":26,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.928651+0000","last_change":"2024-03-26T04:17:17.176526+0000","last_active":"2024-03-26T04:17:50.928651+0000","last_peered":"2024-03-26T04:17:50.928651+0000","last_clean":"2024-03-26T04:17:50.928651+0000","last_became_active":"2024-03-26T04:17:17.176224+0000","last_became_peered":"2024-03-26T04:17:17.176224+0000","last_unstale":"2024-03-26T04:17:50.928651+0000","last_undegraded":"2024-03-26T04:17:50.928651+0000","last_fullsized":"2024-03-26T04:17:50.928651+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:56:50.803919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.039988+0000","last_change":"2024-03-26T04:17:15.168756+0000","last_active":"2024-03-26T04:17:51.039988+0000","last_peered":"2024-03-26T04:17:51.039988+0000","last_clean":"2024-03-26T04:17:51.039988+0000","last_became_active":"2024-03-26T04:17:15.168402+0000","last_became_peered":"2024-03-26T04:17:15.168402+0000","last_unstale":"2024-03-26T04:17:51.039988+0000","last_undegraded":"2024-03-26T04:17:51.039988+0000","last_fullsized":"2024-03-26T04:17:51.039988+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:55:28.119084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.032026+0000","last_change":"2024-03-26T04:17:11.137375+0000","last_active":"2024-03-26T04:17:51.032026+0000","last_peered":"2024-03-26T04:17:51.032026+0000","last_clean":"2024-03-26T04:17:51.032026+0000","last_became_active":"2024-03-26T04:17:11.137073+0000","last_became_peered":"2024-03-26T04:17:11.137073+0000","last_unstale":"2024-03-26T04:17:51.032026+0000","last_undegraded":"2024-03-26T04:17:51.032026+0000","last_fullsized":"2024-03-26T04:17:51.032026+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:10:09.759028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"68'15","reported_seq":53,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.929501+0000","last_change":"2024-03-26T04:17:13.144023+0000","last_active":"2024-03-26T04:17:50.929501+0000","last_peered":"2024-03-26T04:17:50.929501+0000","last_clean":"2024-03-26T04:17:50.929501+0000","last_became_active":"2024-03-26T04:17:13.143687+0000","last_became_peered":"2024-03-26T04:17:13.143687+0000","last_unstale":"2024-03-26T04:17:50.929501+0000","last_undegraded":"2024-03-26T04:17:50.929501+0000","last_fullsized":"2024-03-26T04:17:50.929501+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:00:31.849325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.330978+0000","last_change":"2024-03-26T04:17:17.177350+0000","last_active":"2024-03-26T04:17:51.330978+0000","last_peered":"2024-03-26T04:17:51.330978+0000","last_clean":"2024-03-26T04:17:51.330978+0000","last_became_active":"2024-03-26T04:17:17.177019+0000","last_became_peered":"2024-03-26T04:17:17.177019+0000","last_unstale":"2024-03-26T04:17:51.330978+0000","last_undegraded":"2024-03-26T04:17:51.330978+0000","last_fullsized":"2024-03-26T04:17:51.330978+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:07:32.652282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"64'2","reported_seq":40,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.031773+0000","last_change":"2024-03-26T04:17:14.155484+0000","last_active":"2024-03-26T04:17:51.031773+0000","last_peered":"2024-03-26T04:17:51.031773+0000","last_clean":"2024-03-26T04:17:51.031773+0000","last_became_active":"2024-03-26T04:17:10.129516+0000","last_became_peered":"2024-03-26T04:17:10.129516+0000","last_unstale":"2024-03-26T04:17:51.031773+0000","last_undegraded":"2024-03-26T04:17:51.031773+0000","last_fullsized":"2024-03-26T04:17:51.031773+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:09.109486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:09.109486+0000","last_clean_scrub_stamp":"2024-03-26T04:17:09.109486+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T16:09:47.099491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015728389999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.145107+0000","last_change":"2024-03-26T04:17:11.137658+0000","last_active":"2024-03-26T04:17:51.145107+0000","last_peered":"2024-03-26T04:17:51.145107+0000","last_clean":"2024-03-26T04:17:51.145107+0000","last_became_active":"2024-03-26T04:17:11.137368+0000","last_became_peered":"2024-03-26T04:17:11.137368+0000","last_unstale":"2024-03-26T04:17:51.145107+0000","last_undegraded":"2024-03-26T04:17:51.145107+0000","last_fullsized":"2024-03-26T04:17:51.145107+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:01:39.556213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"68'34","reported_seq":103,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:18:31.561402+0000","last_change":"2024-03-26T04:17:13.154441+0000","last_active":"2024-03-26T04:18:31.561402+0000","last_peered":"2024-03-26T04:18:31.561402+0000","last_clean":"2024-03-26T04:18:31.561402+0000","last_became_active":"2024-03-26T04:17:13.154106+0000","last_became_peered":"2024-03-26T04:17:13.154106+0000","last_unstale":"2024-03-26T04:18:31.561402+0000","last_undegraded":"2024-03-26T04:18:31.561402+0000","last_fullsized":"2024-03-26T04:18:31.561402+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:08:27.442392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":57,"num_read_kb":41,"num_write":32,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.929449+0000","last_change":"2024-03-26T04:17:15.169344+0000","last_active":"2024-03-26T04:17:50.929449+0000","last_peered":"2024-03-26T04:17:50.929449+0000","last_clean":"2024-03-26T04:17:50.929449+0000","last_became_active":"2024-03-26T04:17:15.168809+0000","last_became_peered":"2024-03-26T04:17:15.168809+0000","last_unstale":"2024-03-26T04:17:50.929449+0000","last_undegraded":"2024-03-26T04:17:50.929449+0000","last_fullsized":"2024-03-26T04:17:50.929449+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:21:03.151162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.933037+0000","last_change":"2024-03-26T04:17:17.257393+0000","last_active":"2024-03-26T04:17:50.933037+0000","last_peered":"2024-03-26T04:17:50.933037+0000","last_clean":"2024-03-26T04:17:50.933037+0000","last_became_active":"2024-03-26T04:17:17.257078+0000","last_became_peered":"2024-03-26T04:17:17.257078+0000","last_unstale":"2024-03-26T04:17:50.933037+0000","last_undegraded":"2024-03-26T04:17:50.933037+0000","last_fullsized":"2024-03-26T04:17:50.933037+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:28:06.403543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.240493+0000","last_change":"2024-03-26T04:17:11.507506+0000","last_active":"2024-03-26T04:17:51.240493+0000","last_peered":"2024-03-26T04:17:51.240493+0000","last_clean":"2024-03-26T04:17:51.240493+0000","last_became_active":"2024-03-26T04:17:11.507085+0000","last_became_peered":"2024-03-26T04:17:11.507085+0000","last_unstale":"2024-03-26T04:17:51.240493+0000","last_undegraded":"2024-03-26T04:17:51.240493+0000","last_fullsized":"2024-03-26T04:17:51.240493+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:41:09.015342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":38,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.039947+0000","last_change":"2024-03-26T04:17:14.146806+0000","last_active":"2024-03-26T04:17:51.039947+0000","last_peered":"2024-03-26T04:17:51.039947+0000","last_clean":"2024-03-26T04:17:51.039947+0000","last_became_active":"2024-03-26T04:17:10.130110+0000","last_became_peered":"2024-03-26T04:17:10.130110+0000","last_unstale":"2024-03-26T04:17:51.039947+0000","last_undegraded":"2024-03-26T04:17:51.039947+0000","last_fullsized":"2024-03-26T04:17:51.039947+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:09.109486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:09.109486+0000","last_clean_scrub_stamp":"2024-03-26T04:17:09.109486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:56:41.645772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00051742100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"68'19","reported_seq":64,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.237837+0000","last_change":"2024-03-26T04:17:13.145869+0000","last_active":"2024-03-26T04:17:51.237837+0000","last_peered":"2024-03-26T04:17:51.237837+0000","last_clean":"2024-03-26T04:17:51.237837+0000","last_became_active":"2024-03-26T04:17:13.144709+0000","last_became_peered":"2024-03-26T04:17:13.144709+0000","last_unstale":"2024-03-26T04:17:51.237837+0000","last_undegraded":"2024-03-26T04:17:51.237837+0000","last_fullsized":"2024-03-26T04:17:51.237837+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:12:58.044252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":39,"num_read_kb":27,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.040400+0000","last_change":"2024-03-26T04:17:15.171150+0000","last_active":"2024-03-26T04:17:51.040400+0000","last_peered":"2024-03-26T04:17:51.040400+0000","last_clean":"2024-03-26T04:17:51.040400+0000","last_became_active":"2024-03-26T04:17:15.170576+0000","last_became_peered":"2024-03-26T04:17:15.170576+0000","last_unstale":"2024-03-26T04:17:51.040400+0000","last_undegraded":"2024-03-26T04:17:51.040400+0000","last_fullsized":"2024-03-26T04:17:51.040400+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:20:01.340563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.032225+0000","last_change":"2024-03-26T04:17:17.168363+0000","last_active":"2024-03-26T04:17:51.032225+0000","last_peered":"2024-03-26T04:17:51.032225+0000","last_clean":"2024-03-26T04:17:51.032225+0000","last_became_active":"2024-03-26T04:17:17.168095+0000","last_became_peered":"2024-03-26T04:17:17.168095+0000","last_unstale":"2024-03-26T04:17:51.032225+0000","last_undegraded":"2024-03-26T04:17:51.032225+0000","last_fullsized":"2024-03-26T04:17:51.032225+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:34:15.154296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"70'128","reported_seq":273,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:53.022931+0000","last_change":"2024-03-26T04:17:01.279536+0000","last_active":"2024-03-26T04:17:53.022931+0000","last_peered":"2024-03-26T04:17:53.022931+0000","last_clean":"2024-03-26T04:17:53.022931+0000","last_became_active":"2024-03-26T04:17:01.279151+0000","last_became_peered":"2024-03-26T04:17:01.279151+0000","last_unstale":"2024-03-26T04:17:53.022931+0000","last_undegraded":"2024-03-26T04:17:53.022931+0000","last_fullsized":"2024-03-26T04:17:53.022931+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:14:58.099724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:14:58.099724+0000","last_clean_scrub_stamp":"2024-03-26T04:14:58.099724+0000","objects_scrubbed":0,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:21:39.312605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.329926+0000","last_change":"2024-03-26T04:17:11.138651+0000","last_active":"2024-03-26T04:17:51.329926+0000","last_peered":"2024-03-26T04:17:51.329926+0000","last_clean":"2024-03-26T04:17:51.329926+0000","last_became_active":"2024-03-26T04:17:11.138304+0000","last_became_peered":"2024-03-26T04:17:11.138304+0000","last_unstale":"2024-03-26T04:17:51.329926+0000","last_undegraded":"2024-03-26T04:17:51.329926+0000","last_fullsized":"2024-03-26T04:17:51.329926+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:48:47.763631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"68'16","reported_seq":65,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:18:31.561816+0000","last_change":"2024-03-26T04:17:13.893740+0000","last_active":"2024-03-26T04:18:31.561816+0000","last_peered":"2024-03-26T04:18:31.561816+0000","last_clean":"2024-03-26T04:18:31.561816+0000","last_became_active":"2024-03-26T04:17:13.893175+0000","last_became_peered":"2024-03-26T04:17:13.893175+0000","last_unstale":"2024-03-26T04:18:31.561816+0000","last_undegraded":"2024-03-26T04:18:31.561816+0000","last_fullsized":"2024-03-26T04:18:31.561816+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:08:28.719073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.932285+0000","last_change":"2024-03-26T04:17:15.167344+0000","last_active":"2024-03-26T04:17:50.932285+0000","last_peered":"2024-03-26T04:17:50.932285+0000","last_clean":"2024-03-26T04:17:50.932285+0000","last_became_active":"2024-03-26T04:17:15.167045+0000","last_became_peered":"2024-03-26T04:17:15.167045+0000","last_unstale":"2024-03-26T04:17:50.932285+0000","last_undegraded":"2024-03-26T04:17:50.932285+0000","last_fullsized":"2024-03-26T04:17:50.932285+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:50:27.618263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"68'1","reported_seq":18,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.241035+0000","last_change":"2024-03-26T04:17:17.170240+0000","last_active":"2024-03-26T04:17:51.241035+0000","last_peered":"2024-03-26T04:17:51.241035+0000","last_clean":"2024-03-26T04:17:51.241035+0000","last_became_active":"2024-03-26T04:17:17.169919+0000","last_became_peered":"2024-03-26T04:17:17.169919+0000","last_unstale":"2024-03-26T04:17:51.241035+0000","last_undegraded":"2024-03-26T04:17:51.241035+0000","last_fullsized":"2024-03-26T04:17:51.241035+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:04:03.612791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.928875+0000","last_change":"2024-03-26T04:17:11.135087+0000","last_active":"2024-03-26T04:17:50.928875+0000","last_peered":"2024-03-26T04:17:50.928875+0000","last_clean":"2024-03-26T04:17:50.928875+0000","last_became_active":"2024-03-26T04:17:11.134742+0000","last_became_peered":"2024-03-26T04:17:11.134742+0000","last_unstale":"2024-03-26T04:17:50.928875+0000","last_undegraded":"2024-03-26T04:17:50.928875+0000","last_fullsized":"2024-03-26T04:17:50.928875+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:58:33.967014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"68'5","reported_seq":120,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:18:35.803397+0000","last_change":"2024-03-26T04:17:14.157838+0000","last_active":"2024-03-26T04:18:35.803397+0000","last_peered":"2024-03-26T04:18:35.803397+0000","last_clean":"2024-03-26T04:18:35.803397+0000","last_became_active":"2024-03-26T04:17:10.132378+0000","last_became_peered":"2024-03-26T04:17:10.132378+0000","last_unstale":"2024-03-26T04:18:35.803397+0000","last_undegraded":"2024-03-26T04:18:35.803397+0000","last_fullsized":"2024-03-26T04:18:35.803397+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":59,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:09.109486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:09.109486+0000","last_clean_scrub_stamp":"2024-03-26T04:17:09.109486+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:25:16.913920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0022493809999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":79,"num_read_kb":74,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"68'12","reported_seq":43,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.930119+0000","last_change":"2024-03-26T04:17:13.146942+0000","last_active":"2024-03-26T04:17:50.930119+0000","last_peered":"2024-03-26T04:17:50.930119+0000","last_clean":"2024-03-26T04:17:50.930119+0000","last_became_active":"2024-03-26T04:17:13.146602+0000","last_became_peered":"2024-03-26T04:17:13.146602+0000","last_unstale":"2024-03-26T04:17:50.930119+0000","last_undegraded":"2024-03-26T04:17:50.930119+0000","last_fullsized":"2024-03-26T04:17:50.930119+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:02:06.164330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.032466+0000","last_change":"2024-03-26T04:17:15.169662+0000","last_active":"2024-03-26T04:17:51.032466+0000","last_peered":"2024-03-26T04:17:51.032466+0000","last_clean":"2024-03-26T04:17:51.032466+0000","last_became_active":"2024-03-26T04:17:15.169284+0000","last_became_peered":"2024-03-26T04:17:15.169284+0000","last_unstale":"2024-03-26T04:17:51.032466+0000","last_undegraded":"2024-03-26T04:17:51.032466+0000","last_fullsized":"2024-03-26T04:17:51.032466+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T04:22:34.956409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.239503+0000","last_change":"2024-03-26T04:17:17.257336+0000","last_active":"2024-03-26T04:17:51.239503+0000","last_peered":"2024-03-26T04:17:51.239503+0000","last_clean":"2024-03-26T04:17:51.239503+0000","last_became_active":"2024-03-26T04:17:17.256133+0000","last_became_peered":"2024-03-26T04:17:17.256133+0000","last_unstale":"2024-03-26T04:17:51.239503+0000","last_undegraded":"2024-03-26T04:17:51.239503+0000","last_fullsized":"2024-03-26T04:17:51.239503+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:10:09.000458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.241247+0000","last_change":"2024-03-26T04:17:11.507607+0000","last_active":"2024-03-26T04:17:51.241247+0000","last_peered":"2024-03-26T04:17:51.241247+0000","last_clean":"2024-03-26T04:17:51.241247+0000","last_became_active":"2024-03-26T04:17:11.507267+0000","last_became_peered":"2024-03-26T04:17:11.507267+0000","last_unstale":"2024-03-26T04:17:51.241247+0000","last_undegraded":"2024-03-26T04:17:51.241247+0000","last_fullsized":"2024-03-26T04:17:51.241247+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:18:02.826317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"68'25","reported_seq":73,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.929776+0000","last_change":"2024-03-26T04:17:13.892595+0000","last_active":"2024-03-26T04:17:50.929776+0000","last_peered":"2024-03-26T04:17:50.929776+0000","last_clean":"2024-03-26T04:17:50.929776+0000","last_became_active":"2024-03-26T04:17:13.892263+0000","last_became_peered":"2024-03-26T04:17:13.892263+0000","last_unstale":"2024-03-26T04:17:50.929776+0000","last_undegraded":"2024-03-26T04:17:50.929776+0000","last_fullsized":"2024-03-26T04:17:50.929776+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:43:22.950743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":28,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.330728+0000","last_change":"2024-03-26T04:17:15.167661+0000","last_active":"2024-03-26T04:17:51.330728+0000","last_peered":"2024-03-26T04:17:51.330728+0000","last_clean":"2024-03-26T04:17:51.330728+0000","last_became_active":"2024-03-26T04:17:15.167369+0000","last_became_peered":"2024-03-26T04:17:15.167369+0000","last_unstale":"2024-03-26T04:17:51.330728+0000","last_undegraded":"2024-03-26T04:17:51.330728+0000","last_fullsized":"2024-03-26T04:17:51.330728+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:13:55.556911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.928368+0000","last_change":"2024-03-26T04:17:17.180260+0000","last_active":"2024-03-26T04:17:50.928368+0000","last_peered":"2024-03-26T04:17:50.928368+0000","last_clean":"2024-03-26T04:17:50.928368+0000","last_became_active":"2024-03-26T04:17:17.179927+0000","last_became_peered":"2024-03-26T04:17:17.179927+0000","last_unstale":"2024-03-26T04:17:50.928368+0000","last_undegraded":"2024-03-26T04:17:50.928368+0000","last_fullsized":"2024-03-26T04:17:50.928368+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:48:29.157699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.032199+0000","last_change":"2024-03-26T04:17:11.507753+0000","last_active":"2024-03-26T04:17:51.032199+0000","last_peered":"2024-03-26T04:17:51.032199+0000","last_clean":"2024-03-26T04:17:51.032199+0000","last_became_active":"2024-03-26T04:17:11.507399+0000","last_became_peered":"2024-03-26T04:17:11.507399+0000","last_unstale":"2024-03-26T04:17:51.032199+0000","last_undegraded":"2024-03-26T04:17:51.032199+0000","last_fullsized":"2024-03-26T04:17:51.032199+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:30:25.517263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"68'16","reported_seq":52,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.239985+0000","last_change":"2024-03-26T04:17:13.140889+0000","last_active":"2024-03-26T04:17:51.239985+0000","last_peered":"2024-03-26T04:17:51.239985+0000","last_clean":"2024-03-26T04:17:51.239985+0000","last_became_active":"2024-03-26T04:17:13.140546+0000","last_became_peered":"2024-03-26T04:17:13.140546+0000","last_unstale":"2024-03-26T04:17:51.239985+0000","last_undegraded":"2024-03-26T04:17:51.239985+0000","last_fullsized":"2024-03-26T04:17:51.239985+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:49:44.183611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":18,"num_write":16,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"68'8","reported_seq":47,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:18:31.561388+0000","last_change":"2024-03-26T04:17:15.168504+0000","last_active":"2024-03-26T04:18:31.561388+0000","last_peered":"2024-03-26T04:18:31.561388+0000","last_clean":"2024-03-26T04:18:31.561388+0000","last_became_active":"2024-03-26T04:17:15.167301+0000","last_became_peered":"2024-03-26T04:17:15.167301+0000","last_unstale":"2024-03-26T04:18:31.561388+0000","last_undegraded":"2024-03-26T04:18:31.561388+0000","last_fullsized":"2024-03-26T04:18:31.561388+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:25:27.460165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.932060+0000","last_change":"2024-03-26T04:17:17.258699+0000","last_active":"2024-03-26T04:17:50.932060+0000","last_peered":"2024-03-26T04:17:50.932060+0000","last_clean":"2024-03-26T04:17:50.932060+0000","last_became_active":"2024-03-26T04:17:17.258405+0000","last_became_peered":"2024-03-26T04:17:17.258405+0000","last_unstale":"2024-03-26T04:17:50.932060+0000","last_undegraded":"2024-03-26T04:17:50.932060+0000","last_fullsized":"2024-03-26T04:17:50.932060+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T07:45:12.145413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"61'1","reported_seq":30,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:50.930256+0000","last_change":"2024-03-26T04:17:11.134146+0000","last_active":"2024-03-26T04:17:50.930256+0000","last_peered":"2024-03-26T04:17:50.930256+0000","last_clean":"2024-03-26T04:17:50.930256+0000","last_became_active":"2024-03-26T04:17:11.133861+0000","last_became_peered":"2024-03-26T04:17:11.133861+0000","last_unstale":"2024-03-26T04:17:50.930256+0000","last_undegraded":"2024-03-26T04:17:50.930256+0000","last_fullsized":"2024-03-26T04:17:50.930256+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:32:03.317648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"68'20","reported_seq":58,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.145758+0000","last_change":"2024-03-26T04:17:13.142252+0000","last_active":"2024-03-26T04:17:51.145758+0000","last_peered":"2024-03-26T04:17:51.145758+0000","last_clean":"2024-03-26T04:17:51.145758+0000","last_became_active":"2024-03-26T04:17:13.141826+0000","last_became_peered":"2024-03-26T04:17:13.141826+0000","last_unstale":"2024-03-26T04:17:51.145758+0000","last_undegraded":"2024-03-26T04:17:51.145758+0000","last_fullsized":"2024-03-26T04:17:51.145758+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:15:16.751157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":30,"num_read_kb":22,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"68'8","reported_seq":47,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:18:31.561460+0000","last_change":"2024-03-26T04:17:15.166276+0000","last_active":"2024-03-26T04:18:31.561460+0000","last_peered":"2024-03-26T04:18:31.561460+0000","last_clean":"2024-03-26T04:18:31.561460+0000","last_became_active":"2024-03-26T04:17:15.164738+0000","last_became_peered":"2024-03-26T04:17:15.164738+0000","last_unstale":"2024-03-26T04:18:31.561460+0000","last_undegraded":"2024-03-26T04:18:31.561460+0000","last_fullsized":"2024-03-26T04:18:31.561460+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:11:35.075712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.145875+0000","last_change":"2024-03-26T04:17:17.174331+0000","last_active":"2024-03-26T04:17:51.145875+0000","last_peered":"2024-03-26T04:17:51.145875+0000","last_clean":"2024-03-26T04:17:51.145875+0000","last_became_active":"2024-03-26T04:17:17.173982+0000","last_became_peered":"2024-03-26T04:17:17.173982+0000","last_unstale":"2024-03-26T04:17:51.145875+0000","last_undegraded":"2024-03-26T04:17:51.145875+0000","last_fullsized":"2024-03-26T04:17:51.145875+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:31:25.172307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.330046+0000","last_change":"2024-03-26T04:17:11.137412+0000","last_active":"2024-03-26T04:17:51.330046+0000","last_peered":"2024-03-26T04:17:51.330046+0000","last_clean":"2024-03-26T04:17:51.330046+0000","last_became_active":"2024-03-26T04:17:11.137113+0000","last_became_peered":"2024-03-26T04:17:11.137113+0000","last_unstale":"2024-03-26T04:17:51.330046+0000","last_undegraded":"2024-03-26T04:17:51.330046+0000","last_fullsized":"2024-03-26T04:17:51.330046+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:20:59.116143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"68'30","reported_seq":81,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.329965+0000","last_change":"2024-03-26T04:17:13.894120+0000","last_active":"2024-03-26T04:17:51.329965+0000","last_peered":"2024-03-26T04:17:51.329965+0000","last_clean":"2024-03-26T04:17:51.329965+0000","last_became_active":"2024-03-26T04:17:13.893795+0000","last_became_peered":"2024-03-26T04:17:13.893795+0000","last_unstale":"2024-03-26T04:17:51.329965+0000","last_undegraded":"2024-03-26T04:17:51.329965+0000","last_fullsized":"2024-03-26T04:17:51.329965+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:36:09.783064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":38,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.145603+0000","last_change":"2024-03-26T04:17:15.170379+0000","last_active":"2024-03-26T04:17:51.145603+0000","last_peered":"2024-03-26T04:17:51.145603+0000","last_clean":"2024-03-26T04:17:51.145603+0000","last_became_active":"2024-03-26T04:17:15.170012+0000","last_became_peered":"2024-03-26T04:17:15.170012+0000","last_unstale":"2024-03-26T04:17:51.145603+0000","last_undegraded":"2024-03-26T04:17:51.145603+0000","last_fullsized":"2024-03-26T04:17:51.145603+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:49:21.109132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.032037+0000","last_change":"2024-03-26T04:17:17.175942+0000","last_active":"2024-03-26T04:17:51.032037+0000","last_peered":"2024-03-26T04:17:51.032037+0000","last_clean":"2024-03-26T04:17:51.032037+0000","last_became_active":"2024-03-26T04:17:17.175378+0000","last_became_peered":"2024-03-26T04:17:17.175378+0000","last_unstale":"2024-03-26T04:17:51.032037+0000","last_undegraded":"2024-03-26T04:17:51.032037+0000","last_fullsized":"2024-03-26T04:17:51.032037+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T09:53:26.000527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"68'8","reported_seq":47,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:18:31.561720+0000","last_change":"2024-03-26T04:17:15.169106+0000","last_active":"2024-03-26T04:18:31.561720+0000","last_peered":"2024-03-26T04:18:31.561720+0000","last_clean":"2024-03-26T04:18:31.561720+0000","last_became_active":"2024-03-26T04:17:15.168175+0000","last_became_peered":"2024-03-26T04:17:15.168175+0000","last_unstale":"2024-03-26T04:18:31.561720+0000","last_undegraded":"2024-03-26T04:18:31.561720+0000","last_fullsized":"2024-03-26T04:18:31.561720+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T08:14:38.599782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.329674+0000","last_change":"2024-03-26T04:17:11.139149+0000","last_active":"2024-03-26T04:17:51.329674+0000","last_peered":"2024-03-26T04:17:51.329674+0000","last_clean":"2024-03-26T04:17:51.329674+0000","last_became_active":"2024-03-26T04:17:11.138456+0000","last_became_peered":"2024-03-26T04:17:11.138456+0000","last_unstale":"2024-03-26T04:17:51.329674+0000","last_undegraded":"2024-03-26T04:17:51.329674+0000","last_fullsized":"2024-03-26T04:17:51.329674+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:35:22.875520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"68'15","reported_seq":50,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.238034+0000","last_change":"2024-03-26T04:17:13.153000+0000","last_active":"2024-03-26T04:17:51.238034+0000","last_peered":"2024-03-26T04:17:51.238034+0000","last_clean":"2024-03-26T04:17:51.238034+0000","last_became_active":"2024-03-26T04:17:13.152569+0000","last_became_peered":"2024-03-26T04:17:13.152569+0000","last_unstale":"2024-03-26T04:17:51.238034+0000","last_undegraded":"2024-03-26T04:17:51.238034+0000","last_fullsized":"2024-03-26T04:17:51.238034+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:08:43.535380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.330964+0000","last_change":"2024-03-26T04:17:17.256897+0000","last_active":"2024-03-26T04:17:51.330964+0000","last_peered":"2024-03-26T04:17:51.330964+0000","last_clean":"2024-03-26T04:17:51.330964+0000","last_became_active":"2024-03-26T04:17:17.256122+0000","last_became_peered":"2024-03-26T04:17:17.256122+0000","last_unstale":"2024-03-26T04:17:51.330964+0000","last_undegraded":"2024-03-26T04:17:51.330964+0000","last_fullsized":"2024-03-26T04:17:51.330964+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:19:22.207774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.032409+0000","last_change":"2024-03-26T04:17:15.168405+0000","last_active":"2024-03-26T04:17:51.032409+0000","last_peered":"2024-03-26T04:17:51.032409+0000","last_clean":"2024-03-26T04:17:51.032409+0000","last_became_active":"2024-03-26T04:17:15.168136+0000","last_became_peered":"2024-03-26T04:17:15.168136+0000","last_unstale":"2024-03-26T04:17:51.032409+0000","last_undegraded":"2024-03-26T04:17:51.032409+0000","last_fullsized":"2024-03-26T04:17:51.032409+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T15:43:05.542786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.145625+0000","last_change":"2024-03-26T04:17:11.507797+0000","last_active":"2024-03-26T04:17:51.145625+0000","last_peered":"2024-03-26T04:17:51.145625+0000","last_clean":"2024-03-26T04:17:51.145625+0000","last_became_active":"2024-03-26T04:17:11.507413+0000","last_became_peered":"2024-03-26T04:17:11.507413+0000","last_unstale":"2024-03-26T04:17:51.145625+0000","last_undegraded":"2024-03-26T04:17:51.145625+0000","last_fullsized":"2024-03-26T04:17:51.145625+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:32:21.104531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"68'17","reported_seq":56,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.146323+0000","last_change":"2024-03-26T04:17:13.144972+0000","last_active":"2024-03-26T04:17:51.146323+0000","last_peered":"2024-03-26T04:17:51.146323+0000","last_clean":"2024-03-26T04:17:51.146323+0000","last_became_active":"2024-03-26T04:17:13.144563+0000","last_became_peered":"2024-03-26T04:17:13.144563+0000","last_unstale":"2024-03-26T04:17:51.146323+0000","last_undegraded":"2024-03-26T04:17:51.146323+0000","last_fullsized":"2024-03-26T04:17:51.146323+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T05:51:42.089112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":29,"num_read_kb":21,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.040549+0000","last_change":"2024-03-26T04:17:17.176526+0000","last_active":"2024-03-26T04:17:51.040549+0000","last_peered":"2024-03-26T04:17:51.040549+0000","last_clean":"2024-03-26T04:17:51.040549+0000","last_became_active":"2024-03-26T04:17:17.174946+0000","last_became_peered":"2024-03-26T04:17:17.174946+0000","last_unstale":"2024-03-26T04:17:51.040549+0000","last_undegraded":"2024-03-26T04:17:51.040549+0000","last_fullsized":"2024-03-26T04:17:51.040549+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T12:33:54.106103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":21,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.241590+0000","last_change":"2024-03-26T04:17:15.171000+0000","last_active":"2024-03-26T04:17:51.241590+0000","last_peered":"2024-03-26T04:17:51.241590+0000","last_clean":"2024-03-26T04:17:51.241590+0000","last_became_active":"2024-03-26T04:17:15.170655+0000","last_became_peered":"2024-03-26T04:17:15.170655+0000","last_unstale":"2024-03-26T04:17:51.241590+0000","last_undegraded":"2024-03-26T04:17:51.241590+0000","last_fullsized":"2024-03-26T04:17:51.241590+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T13:26:50.974086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.330598+0000","last_change":"2024-03-26T04:17:11.140891+0000","last_active":"2024-03-26T04:17:51.330598+0000","last_peered":"2024-03-26T04:17:51.330598+0000","last_clean":"2024-03-26T04:17:51.330598+0000","last_became_active":"2024-03-26T04:17:11.140598+0000","last_became_peered":"2024-03-26T04:17:11.140598+0000","last_unstale":"2024-03-26T04:17:51.330598+0000","last_undegraded":"2024-03-26T04:17:51.330598+0000","last_fullsized":"2024-03-26T04:17:51.330598+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T06:05:09.457124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"68'17","reported_seq":53,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.146545+0000","last_change":"2024-03-26T04:17:13.146593+0000","last_active":"2024-03-26T04:17:51.146545+0000","last_peered":"2024-03-26T04:17:51.146545+0000","last_clean":"2024-03-26T04:17:51.146545+0000","last_became_active":"2024-03-26T04:17:13.146288+0000","last_became_peered":"2024-03-26T04:17:13.146288+0000","last_unstale":"2024-03-26T04:17:51.146545+0000","last_undegraded":"2024-03-26T04:17:51.146545+0000","last_fullsized":"2024-03-26T04:17:51.146545+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T11:53:16.752017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":17,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.146469+0000","last_change":"2024-03-26T04:17:17.175151+0000","last_active":"2024-03-26T04:17:51.146469+0000","last_peered":"2024-03-26T04:17:51.146469+0000","last_clean":"2024-03-26T04:17:51.146469+0000","last_became_active":"2024-03-26T04:17:17.174794+0000","last_became_peered":"2024-03-26T04:17:17.174794+0000","last_unstale":"2024-03-26T04:17:51.146469+0000","last_undegraded":"2024-03-26T04:17:51.146469+0000","last_fullsized":"2024-03-26T04:17:51.146469+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:16.152064+0000","last_clean_scrub_stamp":"2024-03-26T04:17:16.152064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:15:45.052823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"68'8","reported_seq":47,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:18:31.561384+0000","last_change":"2024-03-26T04:17:15.170623+0000","last_active":"2024-03-26T04:18:31.561384+0000","last_peered":"2024-03-26T04:18:31.561384+0000","last_clean":"2024-03-26T04:18:31.561384+0000","last_became_active":"2024-03-26T04:17:15.170254+0000","last_became_peered":"2024-03-26T04:17:15.170254+0000","last_unstale":"2024-03-26T04:18:31.561384+0000","last_undegraded":"2024-03-26T04:18:31.561384+0000","last_fullsized":"2024-03-26T04:18:31.561384+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:14.140708+0000","last_clean_scrub_stamp":"2024-03-26T04:17:14.140708+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T10:29:10.632688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":29,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.329698+0000","last_change":"2024-03-26T04:17:11.140053+0000","last_active":"2024-03-26T04:17:51.329698+0000","last_peered":"2024-03-26T04:17:51.329698+0000","last_clean":"2024-03-26T04:17:51.329698+0000","last_became_active":"2024-03-26T04:17:11.139726+0000","last_became_peered":"2024-03-26T04:17:11.139726+0000","last_unstale":"2024-03-26T04:17:51.329698+0000","last_undegraded":"2024-03-26T04:17:51.329698+0000","last_fullsized":"2024-03-26T04:17:51.329698+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:10.114394+0000","last_clean_scrub_stamp":"2024-03-26T04:17:10.114394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:51:35.051518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"68'10","reported_seq":40,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-03-26T04:17:51.146047+0000","last_change":"2024-03-26T04:17:13.150909+0000","last_active":"2024-03-26T04:17:51.146047+0000","last_peered":"2024-03-26T04:17:51.146047+0000","last_clean":"2024-03-26T04:17:51.146047+0000","last_became_active":"2024-03-26T04:17:13.150582+0000","last_became_peered":"2024-03-26T04:17:13.150582+0000","last_unstale":"2024-03-26T04:17:51.146047+0000","last_undegraded":"2024-03-26T04:17:51.146047+0000","last_fullsized":"2024-03-26T04:17:51.146047+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-26T04:17:12.125899+0000","last_clean_scrub_stamp":"2024-03-26T04:17:12.125899+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-27T14:26:13.619244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":210,"num_object_clones":0,"num_object_copies":630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":210,"num_whiteouts":0,"num_read":986,"num_read_kb":711,"num_write":609,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":587,"ondisk_log_size":587,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1457,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":79,"num_read_kb":74,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234006,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32432,"kb_used_data":1256,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712720,"statfs":{"total":95995035648,"available":95961825280,"internally_reserved":0,"allocated":1286144,"data_stored":859743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495643,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32416,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712736,"statfs":{"total":95995035648,"available":95961841664,"internally_reserved":0,"allocated":1265664,"data_stored":858691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27748,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717404,"statfs":{"total":95995035648,"available":95966621696,"internally_reserved":0,"allocated":745472,"data_stored":270519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738424,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32440,"kb_used_data":1264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712712,"statfs":{"total":95995035648,"available":95961817088,"internally_reserved":0,"allocated":1294336,"data_stored":860360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411375,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27680,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717472,"statfs":{"total":95995035648,"available":95966691328,"internally_reserved":0,"allocated":675840,"data_stored":269711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182442,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31876,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713276,"statfs":{"total":95995035648,"available":95962394624,"internally_reserved":0,"allocated":712704,"data_stored":269227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":30,"seq":128849018916,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31868,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713284,"statfs":{"total":95995035648,"available":95962402816,"internally_reserved":0,"allocated":704512,"data_stored":269326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757278,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27688,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717464,"statfs":{"total":95995035648,"available":95966683136,"internally_reserved":0,"allocated":684032,"data_stored":269537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1013,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1273,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-26T04:18:38.325 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-26T04:18:38.325 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-26T04:18:38.325 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-26T04:18:38.325 DEBUG:teuthology.orchestra.run.smithi016:> sudo cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a shell --fsid d7d59826-eb26-11ee-95d0-87774f69a715 -- ceph health --format=json 2024-03-26T04:18:38.579 INFO:teuthology.orchestra.run.smithi016.stderr:Inferring config /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/mon.a/config 2024-03-26T04:18:38.946 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:18:38 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:18:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:18:38.947 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:18:38 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-alertmanager-a[72105]: ts=2024-03-26T04:18:38.686Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000289791s 2024-03-26T04:18:40.280 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:40 smithi016 ceph-mon[37215]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:18:40.280 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:40 smithi016 ceph-mon[29101]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:18:40.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:40 smithi183 ceph-mon[33745]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:18:40.314 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.312+0000 7f143bac0700 1 -- 172.21.15.16:0/804441509 >> v1:172.21.15.16:6790/0 conn(0x7f143410d7a0 legacy=0x7f143410fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:40.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.312+0000 7f143bac0700 1 -- 172.21.15.16:0/804441509 shutdown_connections 2024-03-26T04:18:40.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.312+0000 7f143bac0700 1 -- 172.21.15.16:0/804441509 >> 172.21.15.16:0/804441509 conn(0x7f14340fc820 msgr2=0x7f14340fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:18:40.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.312+0000 7f143bac0700 1 -- 172.21.15.16:0/804441509 shutdown_connections 2024-03-26T04:18:40.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.312+0000 7f143bac0700 1 -- 172.21.15.16:0/804441509 wait complete. 2024-03-26T04:18:40.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.313+0000 7f143bac0700 1 Processor -- start 2024-03-26T04:18:40.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.313+0000 7f143bac0700 1 -- start start 2024-03-26T04:18:40.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.313+0000 7f143bac0700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f14341a6cf0 con 0x7f1434100f70 2024-03-26T04:18:40.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.313+0000 7f143bac0700 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f14341a7ed0 con 0x7f1434109c00 2024-03-26T04:18:40.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.313+0000 7f143bac0700 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f14341a90d0 con 0x7f143410d7a0 2024-03-26T04:18:40.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.313+0000 7f143a05d700 1 --1- >> v1:172.21.15.16:6790/0 conn(0x7f143410d7a0 0x7f14341a53b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6790/0 says I am v1:172.21.15.16:41106/0 (socket says 172.21.15.16:41106) 2024-03-26T04:18:40.315 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.313+0000 7f143a05d700 1 -- 172.21.15.16:0/457420984 learned_addr learned my addr 172.21.15.16:0/457420984 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:18:40.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.313+0000 7f1426ffd700 1 -- 172.21.15.16:0/457420984 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3703394267 0 0) 0x7f14341a6cf0 con 0x7f1434100f70 2024-03-26T04:18:40.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.313+0000 7f1426ffd700 1 -- 172.21.15.16:0/457420984 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14200049e0 con 0x7f1434100f70 2024-03-26T04:18:40.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.313+0000 7f1426ffd700 1 -- 172.21.15.16:0/457420984 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4174835012 0 0) 0x7f14341a90d0 con 0x7f143410d7a0 2024-03-26T04:18:40.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.313+0000 7f1426ffd700 1 -- 172.21.15.16:0/457420984 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14341a6cf0 con 0x7f143410d7a0 2024-03-26T04:18:40.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.313+0000 7f1426ffd700 1 -- 172.21.15.16:0/457420984 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4150130204 0 0) 0x7f14341a7ed0 con 0x7f1434109c00 2024-03-26T04:18:40.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.313+0000 7f1426ffd700 1 -- 172.21.15.16:0/457420984 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14341a90d0 con 0x7f1434109c00 2024-03-26T04:18:40.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.314+0000 7f1426ffd700 1 -- 172.21.15.16:0/457420984 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2946540338 0 0) 0x7f14200049e0 con 0x7f1434100f70 2024-03-26T04:18:40.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.314+0000 7f1426ffd700 1 -- 172.21.15.16:0/457420984 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14341a7ed0 con 0x7f1434100f70 2024-03-26T04:18:40.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.314+0000 7f1426ffd700 1 -- 172.21.15.16:0/457420984 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3480980161 0 0) 0x7f14341a90d0 con 0x7f1434109c00 2024-03-26T04:18:40.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.314+0000 7f1426ffd700 1 -- 172.21.15.16:0/457420984 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14200049e0 con 0x7f1434109c00 2024-03-26T04:18:40.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.314+0000 7f1426ffd700 1 -- 172.21.15.16:0/457420984 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2666354604 0 0) 0x7f14341a6cf0 con 0x7f143410d7a0 2024-03-26T04:18:40.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.314+0000 7f1426ffd700 1 -- 172.21.15.16:0/457420984 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14341a90d0 con 0x7f143410d7a0 2024-03-26T04:18:40.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.314+0000 7f1426ffd700 1 -- 172.21.15.16:0/457420984 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f14300018b0 con 0x7f1434100f70 2024-03-26T04:18:40.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.314+0000 7f1426ffd700 1 -- 172.21.15.16:0/457420984 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f1428001850 con 0x7f1434109c00 2024-03-26T04:18:40.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.314+0000 7f1426ffd700 1 -- 172.21.15.16:0/457420984 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f142c004730 con 0x7f143410d7a0 2024-03-26T04:18:40.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.314+0000 7f1426ffd700 1 -- 172.21.15.16:0/457420984 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1556899271 0 0) 0x7f14341a7ed0 con 0x7f1434100f70 2024-03-26T04:18:40.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.314+0000 7f1426ffd700 1 -- 172.21.15.16:0/457420984 >> v1:172.21.15.16:6790/0 conn(0x7f143410d7a0 legacy=0x7f14341a53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:40.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.314+0000 7f1426ffd700 1 -- 172.21.15.16:0/457420984 >> v1:172.21.15.183:6789/0 conn(0x7f1434109c00 legacy=0x7f14341a1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:40.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.314+0000 7f1426ffd700 1 -- 172.21.15.16:0/457420984 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14341aa2d0 con 0x7f1434100f70 2024-03-26T04:18:40.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.314+0000 7f143bac0700 1 -- 172.21.15.16:0/457420984 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14341a6ee0 con 0x7f1434100f70 2024-03-26T04:18:40.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.314+0000 7f143bac0700 1 -- 172.21.15.16:0/457420984 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f14341a7360 con 0x7f1434100f70 2024-03-26T04:18:40.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.314+0000 7f1426ffd700 1 -- 172.21.15.16:0/457420984 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f1430003b30 con 0x7f1434100f70 2024-03-26T04:18:40.317 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.314+0000 7f1426ffd700 1 -- 172.21.15.16:0/457420984 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f1430004960 con 0x7f1434100f70 2024-03-26T04:18:40.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.315+0000 7f143bac0700 1 -- 172.21.15.16:0/457420984 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1418005120 con 0x7f1434100f70 2024-03-26T04:18:40.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.316+0000 7f1426ffd700 1 -- 172.21.15.16:0/457420984 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 22) v1 ==== 91420+0+0 (unknown 3095259778 0 0) 0x7f14300036a0 con 0x7f1434100f70 2024-03-26T04:18:40.318 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.316+0000 7f1426ffd700 1 -- 172.21.15.16:0/457420984 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(70..70 src has 1..70) v4 ==== 6172+0+0 (unknown 1260882646 0 0) 0x7f1430087600 con 0x7f1434100f70 2024-03-26T04:18:40.321 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.321+0000 7f1426ffd700 1 -- 172.21.15.16:0/457420984 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f1430054940 con 0x7f1434100f70 2024-03-26T04:18:40.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.569+0000 7f143bac0700 1 -- 172.21.15.16:0/457420984 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1418005950 con 0x7f1434100f70 2024-03-26T04:18:40.569 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.569+0000 7f1426ffd700 1 -- 172.21.15.16:0/457420984 <== mon.0 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (unknown 3730786198 0 4185958460) 0x7f1430052cb0 con 0x7f1434100f70 2024-03-26T04:18:40.570 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-03-26T04:18:40.570 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-26T04:18:40.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.571+0000 7f143bac0700 1 -- 172.21.15.16:0/457420984 >> v1:172.21.15.16:6800/3868004154 conn(0x7f142006eb70 legacy=0x7f1420071030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:40.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.571+0000 7f143bac0700 1 -- 172.21.15.16:0/457420984 >> v1:172.21.15.16:6789/0 conn(0x7f1434100f70 legacy=0x7f14341061b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:18:40.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.571+0000 7f143bac0700 1 -- 172.21.15.16:0/457420984 shutdown_connections 2024-03-26T04:18:40.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.571+0000 7f143bac0700 1 -- 172.21.15.16:0/457420984 >> 172.21.15.16:0/457420984 conn(0x7f14340fc820 msgr2=0x7f1434110410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:18:40.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.571+0000 7f143bac0700 1 -- 172.21.15.16:0/457420984 shutdown_connections 2024-03-26T04:18:40.572 INFO:teuthology.orchestra.run.smithi016.stderr:2024-03-26T04:18:40.571+0000 7f143bac0700 1 -- 172.21.15.16:0/457420984 wait complete. 2024-03-26T04:18:41.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:41 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/457420984' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T04:18:41.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:41 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/457420984' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T04:18:41.266 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-26T04:18:41.266 INFO:tasks.cephadm:Setup complete, yielding 2024-03-26T04:18:41.266 INFO:teuthology.run_tasks:Running task workunit... 2024-03-26T04:18:41.280 INFO:tasks.workunit:Pulling workunits from ref 9f04c91b77e4619963d41730d8cf756e3a6152de 2024-03-26T04:18:41.280 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-26T04:18:41.281 DEBUG:teuthology.orchestra.run.smithi016:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-26T04:18:41.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:41 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/457420984' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-26T04:18:41.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:18:41.297 INFO:teuthology.orchestra.run.smithi016.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-26T04:18:41.298 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-03-26T04:18:41.354 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-03-26T04:18:41.354 DEBUG:teuthology.orchestra.run.smithi016:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-03-26T04:18:41.409 INFO:tasks.workunit:timeout=3h 2024-03-26T04:18:41.410 INFO:tasks.workunit:cleanup=True 2024-03-26T04:18:41.410 DEBUG:teuthology.orchestra.run.smithi016:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 9f04c91b77e4619963d41730d8cf756e3a6152de 2024-03-26T04:18:41.466 INFO:tasks.workunit.client.0.smithi016.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-26T04:18:42.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:42 smithi183 ceph-mon[33745]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:18:42.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:42 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:42.292 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:18:41 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:18:42.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:42 smithi016 ceph-mon[37215]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:18:42.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:42 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:42.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:42 smithi016 ceph-mon[29101]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:18:42.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:42 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:43.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:43 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:18:43.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:43 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:18:43.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:43 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:18:44.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:44 smithi183 ceph-mon[33745]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:18:44.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:44 smithi016 ceph-mon[29101]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:18:44.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:44 smithi016 ceph-mon[37215]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:18:46.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:46 smithi183 ceph-mon[33745]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:18:46.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:46 smithi016 ceph-mon[29101]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:18:46.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:46 smithi016 ceph-mon[37215]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:18:46.912 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:18:46 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-alertmanager-a[72105]: ts=2024-03-26T04:18:46.689Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.00250796s 2024-03-26T04:18:48.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:48 smithi183 ceph-mon[33745]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:18:48.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:48 smithi016 ceph-mon[29101]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:18:48.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:48 smithi016 ceph-mon[37215]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:18:49.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:18:48 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:18:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:18:49.987 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:49 smithi183 systemd[1]: Starting Ceph grafana.a for d7d59826-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:18:50.272 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-mon[33745]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:18:50.273 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:49 smithi183 podman[63892]: 2024-03-26 04:18:49.88934231 +0000 UTC m=+0.017108887 image pull a0ea5e3c46d9434bd173d585b7c2638c7451eaf2543d1d22f9e78442615aaffb quay.io/ceph/ceph-grafana:9.4.12 2024-03-26T04:18:50.273 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:49 smithi183 podman[63892]: 2024-03-26 04:18:49.987749112 +0000 UTC m=+0.115515687 container create 05c46305d2371c8551976e74710de8d17db7ac6f5ac3d3f8f6eae337e91be22c (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a, io.buildah.version=1.32.0, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , version=8.5, name=ubi8, vendor=Red Hat, Inc., vcs-type=git, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=239.1651231664, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, distribution-scope=public, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, io.k8s.display-name=Red Hat Universal Base Image 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, architecture=x86_64, description=Ceph Grafana Container, com.redhat.component=ubi8-container, build-date=2022-04-29T11:48:11.055897, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-03-26T04:18:50.273 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 podman[63892]: 2024-03-26 04:18:50.198546084 +0000 UTC m=+0.326312661 container init 05c46305d2371c8551976e74710de8d17db7ac6f5ac3d3f8f6eae337e91be22c (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.32.0, maintainer=Paul Cuzner , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, build-date=2022-04-29T11:48:11.055897, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, architecture=x86_64, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, release=239.1651231664, description=Ceph Grafana Container, vcs-type=git, vendor=Red Hat, Inc., io.openshift.tags=base rhel8, version=8.5, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8) 2024-03-26T04:18:50.273 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 podman[63892]: 2024-03-26 04:18:50.201297299 +0000 UTC m=+0.329063874 container start 05c46305d2371c8551976e74710de8d17db7ac6f5ac3d3f8f6eae337e91be22c (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, description=Ceph Grafana Container, io.buildah.version=1.32.0, vendor=Red Hat, Inc., architecture=x86_64, release=239.1651231664, vcs-type=git, build-date=2022-04-29T11:48:11.055897, distribution-scope=public, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=8.5, com.redhat.component=ubi8-container, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.openshift.expose-services=, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, maintainer=Paul Cuzner ) 2024-03-26T04:18:50.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:50 smithi016 ceph-mon[37215]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:18:50.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:50 smithi016 ceph-mon[29101]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:18:50.526 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 bash[63892]: 05c46305d2371c8551976e74710de8d17db7ac6f5ac3d3f8f6eae337e91be22c 2024-03-26T04:18:50.526 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 systemd[1]: Started Ceph grafana.a for d7d59826-eb26-11ee-95d0-87774f69a715. 2024-03-26T04:18:50.531 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=settings t=2024-03-26T04:18:50.362055996Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-26T04:18:50.532 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=settings t=2024-03-26T04:18:50.362276061Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-26T04:18:50.532 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=settings t=2024-03-26T04:18:50.362301008Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-26T04:18:50.532 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=settings t=2024-03-26T04:18:50.362314122Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-26T04:18:50.532 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=settings t=2024-03-26T04:18:50.362324526Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-26T04:18:50.532 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=settings t=2024-03-26T04:18:50.362334758Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-26T04:18:50.532 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=settings t=2024-03-26T04:18:50.362344742Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-26T04:18:50.532 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=settings t=2024-03-26T04:18:50.362355079Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-26T04:18:50.532 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=settings t=2024-03-26T04:18:50.362366056Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-26T04:18:50.532 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=settings t=2024-03-26T04:18:50.362376342Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-26T04:18:50.532 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=settings t=2024-03-26T04:18:50.362385965Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-26T04:18:50.532 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=settings t=2024-03-26T04:18:50.36239547Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-26T04:18:50.532 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=settings t=2024-03-26T04:18:50.362405176Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-26T04:18:50.532 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=settings t=2024-03-26T04:18:50.362415019Z level=info msg="App mode production" 2024-03-26T04:18:50.533 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=sqlstore t=2024-03-26T04:18:50.362483042Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-26T04:18:50.533 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=sqlstore t=2024-03-26T04:18:50.362504707Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-26T04:18:50.533 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.363368911Z level=info msg="Starting DB migrations" 2024-03-26T04:18:50.533 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.363447077Z level=info msg="Executing migration" id="create migration_log table" 2024-03-26T04:18:50.533 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.363970533Z level=info msg="Executing migration" id="create user table" 2024-03-26T04:18:50.533 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.364458934Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-26T04:18:50.533 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.365287278Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-26T04:18:50.533 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.366165265Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-26T04:18:50.533 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.366992977Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-26T04:18:50.533 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.367397316Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-26T04:18:50.533 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.368252164Z level=info msg="Executing migration" id="create user table v2" 2024-03-26T04:18:50.533 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.369216222Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-26T04:18:50.533 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.369935699Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-26T04:18:50.533 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.37036914Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-26T04:18:50.534 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.37132056Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-26T04:18:50.534 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.373960537Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-26T04:18:50.534 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.376700531Z level=info msg="Executing migration" id="Update user table charset" 2024-03-26T04:18:50.534 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.379577527Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-26T04:18:50.534 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.382306503Z level=info msg="Executing migration" id="Add missing user data" 2024-03-26T04:18:50.534 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.385202896Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-26T04:18:50.534 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.388038929Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-26T04:18:50.535 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.390733197Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-26T04:18:50.535 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.393482823Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-26T04:18:50.537 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.396260366Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-26T04:18:50.537 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.399244579Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-26T04:18:50.538 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.401862236Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-26T04:18:50.538 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.404575179Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-26T04:18:50.538 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.407288861Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-26T04:18:50.538 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.410104911Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-26T04:18:50.538 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.412709718Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-26T04:18:50.538 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.413097569Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-26T04:18:50.538 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.416012696Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-26T04:18:50.538 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.418640291Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-26T04:18:50.538 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.421366855Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-26T04:18:50.538 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.422185862Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-26T04:18:50.538 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.424933427Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-26T04:18:50.538 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.427659927Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-26T04:18:50.538 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.430350867Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-26T04:18:50.539 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.430971157Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-26T04:18:50.539 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.433775959Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-26T04:18:50.539 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.436212355Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-26T04:18:50.539 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.438852154Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-26T04:18:50.539 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.441413838Z level=info msg="Executing migration" id="create star table" 2024-03-26T04:18:50.539 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.444272169Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-26T04:18:50.539 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.446931615Z level=info msg="Executing migration" id="create org table v1" 2024-03-26T04:18:50.539 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.449392199Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-26T04:18:50.539 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.449842865Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-26T04:18:50.539 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.450249533Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-26T04:18:50.539 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.450678441Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-26T04:18:50.539 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.451118835Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-26T04:18:50.539 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.451567573Z level=info msg="Executing migration" id="Update org table charset" 2024-03-26T04:18:50.539 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.451854566Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-26T04:18:50.540 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.452147665Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-26T04:18:50.540 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.452495668Z level=info msg="Executing migration" id="create dashboard table" 2024-03-26T04:18:50.540 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.452890403Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-26T04:18:50.541 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.453307352Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-26T04:18:50.541 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.453725863Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-26T04:18:50.541 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.454133194Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-26T04:18:50.541 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.456915661Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-26T04:18:50.541 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.459581361Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-26T04:18:50.541 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.460759824Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-26T04:18:50.541 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.461611179Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-26T04:18:50.541 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.462473973Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-26T04:18:50.541 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.463408784Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-26T04:18:50.541 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.464014684Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-26T04:18:50.541 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.464457743Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-26T04:18:50.541 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.464822201Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-26T04:18:50.542 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.465366452Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-26T04:18:50.542 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.466132329Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-26T04:18:50.542 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.467076228Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-26T04:18:50.542 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.467776034Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-26T04:18:50.542 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.468630247Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-26T04:18:50.542 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.469534127Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-26T04:18:50.542 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.47020758Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-26T04:18:50.542 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.470584924Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-26T04:18:50.542 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.470886758Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-26T04:18:50.542 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.471453366Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-26T04:18:50.542 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.472309687Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-26T04:18:50.542 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.473202061Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-26T04:18:50.542 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.473941463Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-26T04:18:50.542 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.474763851Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-26T04:18:50.543 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.475558843Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-26T04:18:50.543 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.47645474Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-26T04:18:50.543 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.477505824Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-26T04:18:50.543 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.478163076Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-26T04:18:50.543 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.479066745Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-26T04:18:50.543 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.480060244Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-26T04:18:50.543 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.481031502Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-26T04:18:50.544 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.481908586Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-26T04:18:50.544 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.482627745Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-26T04:18:50.544 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.483470106Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-26T04:18:50.544 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.484307931Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-26T04:18:50.544 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.485284654Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-26T04:18:50.544 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.486029829Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-26T04:18:50.544 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.486998206Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-26T04:18:50.544 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.487740296Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-26T04:18:50.544 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.488732697Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-26T04:18:50.544 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.489380467Z level=info msg="Executing migration" id="create data_source table" 2024-03-26T04:18:50.544 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.490393797Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-26T04:18:50.544 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.490853484Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-26T04:18:50.544 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.493154907Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-26T04:18:50.545 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.496069038Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-26T04:18:50.545 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.498791423Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-26T04:18:50.545 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.501781942Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-26T04:18:50.545 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.504368336Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-26T04:18:50.545 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.507202395Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-26T04:18:50.545 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.509920678Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-26T04:18:50.545 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.512667749Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-26T04:18:50.545 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.515303483Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-26T04:18:50.545 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.518019543Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-26T04:18:50.545 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.520605381Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-26T04:18:50.545 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.523358917Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-26T04:18:50.545 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.526214691Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-26T04:18:50.545 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.529027184Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-26T04:18:50.777 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.531813048Z level=info msg="Executing migration" id="Add uid column" 2024-03-26T04:18:50.777 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.534454569Z level=info msg="Executing migration" id="Update uid value" 2024-03-26T04:18:50.777 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.53725238Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-26T04:18:50.777 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.540053515Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-26T04:18:50.778 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.542843589Z level=info msg="Executing migration" id="create api_key table" 2024-03-26T04:18:50.778 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.545226287Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-26T04:18:50.778 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.547903996Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-26T04:18:50.778 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.550281047Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-26T04:18:50.778 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.553017531Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-26T04:18:50.778 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.555684451Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-26T04:18:50.778 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.558360396Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-26T04:18:50.778 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.560984421Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-26T04:18:50.778 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.563710069Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-26T04:18:50.778 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.564322083Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-26T04:18:50.778 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.566752928Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-26T04:18:50.778 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.569542337Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-26T04:18:50.778 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.57217301Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-26T04:18:50.779 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.572499962Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-26T04:18:50.779 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.572825115Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-26T04:18:50.779 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.573085128Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-26T04:18:50.779 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.573632331Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-26T04:18:50.779 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.5754845Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-26T04:18:50.779 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.575837288Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-26T04:18:50.779 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.576378527Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-26T04:18:50.779 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.576913848Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-26T04:18:50.779 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.577704515Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-26T04:18:50.779 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.578102687Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-26T04:18:50.779 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.578520312Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-26T04:18:50.779 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.579372804Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-26T04:18:50.779 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.579816353Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-26T04:18:50.780 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.580649181Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-26T04:18:50.780 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.581522954Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-26T04:18:50.780 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.582227615Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-26T04:18:50.780 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.582786319Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-26T04:18:50.780 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.583599784Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-26T04:18:50.780 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.584620301Z level=info msg="Executing migration" id="create quota table v1" 2024-03-26T04:18:50.780 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.585271901Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-26T04:18:50.780 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.585732532Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-26T04:18:50.782 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.586034899Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-26T04:18:50.782 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.586521793Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-26T04:18:50.782 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.587299315Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-26T04:18:50.782 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.588202022Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-26T04:18:50.782 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.588528331Z level=info msg="Executing migration" id="create session table" 2024-03-26T04:18:50.782 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.589019652Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-26T04:18:50.782 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.5898563Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-26T04:18:50.782 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.590197176Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-26T04:18:50.782 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.590626153Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-26T04:18:50.782 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.591549479Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-26T04:18:50.783 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.592407947Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-26T04:18:50.783 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.592928889Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-26T04:18:50.783 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.593218879Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-26T04:18:50.783 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.593530879Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-26T04:18:50.783 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.59392786Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-26T04:18:50.783 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.5941963Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-26T04:18:50.783 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.59479614Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-26T04:18:50.783 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.595091286Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-26T04:18:50.783 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.595706565Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-26T04:18:50.783 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.596343126Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-26T04:18:50.783 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.597173059Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-26T04:18:50.783 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.597612111Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-26T04:18:50.783 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.598488896Z level=info msg="Executing migration" id="create alert table v1" 2024-03-26T04:18:50.784 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.598976695Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-26T04:18:50.784 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.599950076Z level=info msg="Executing migration" id="add index alert state" 2024-03-26T04:18:50.784 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.600635245Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-26T04:18:50.784 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.601049832Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-26T04:18:50.784 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.601478906Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-26T04:18:50.784 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.60190742Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-26T04:18:50.784 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.602347791Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-26T04:18:50.784 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.603763175Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-26T04:18:50.784 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.604165749Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-26T04:18:50.784 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.606805073Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-26T04:18:50.784 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.609478463Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-26T04:18:50.784 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.612196722Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-26T04:18:50.784 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.612735154Z level=info msg="Executing migration" id="Add column is_default" 2024-03-26T04:18:50.784 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.615450914Z level=info msg="Executing migration" id="Add column frequency" 2024-03-26T04:18:50.785 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.618043199Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-26T04:18:50.785 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.618814709Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-26T04:18:50.785 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.619566332Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-26T04:18:50.785 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.620343141Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-26T04:18:50.785 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.621240878Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-26T04:18:50.785 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.621893896Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-26T04:18:50.785 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.622327425Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-26T04:18:50.785 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.623121228Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-26T04:18:50.785 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.62410583Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-26T04:18:50.785 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.62471479Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-26T04:18:50.785 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.625108314Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-26T04:18:50.785 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.625759178Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-26T04:18:50.785 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.626415168Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-26T04:18:50.786 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.627342782Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-26T04:18:50.786 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.628019588Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-26T04:18:50.786 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.628436706Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-26T04:18:50.786 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.629272708Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-26T04:18:50.786 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.630193743Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-26T04:18:50.786 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.630945379Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-26T04:18:50.786 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.631383258Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-26T04:18:50.786 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.632413409Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-26T04:18:50.786 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.633312959Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-26T04:18:50.786 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.634025668Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-26T04:18:50.786 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.634455795Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-26T04:18:50.786 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.635475252Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-26T04:18:50.786 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.63620278Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-26T04:18:50.786 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.637168881Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-26T04:18:50.787 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.637629593Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-26T04:18:50.787 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.638452051Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-26T04:18:50.787 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.639322132Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-26T04:18:50.787 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.640187499Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-26T04:18:50.787 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.640621029Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-26T04:18:50.787 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.64154935Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-26T04:18:50.787 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.642297233Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-26T04:18:50.787 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.643890449Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-26T04:18:50.787 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.644309117Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-26T04:18:50.787 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.647360617Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-26T04:18:50.787 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.649880207Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-26T04:18:50.787 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.652881489Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-26T04:18:50.787 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.653242516Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-26T04:18:50.788 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.656216697Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-26T04:18:50.788 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.65903085Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-26T04:18:50.788 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.661817744Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-26T04:18:50.788 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.662263831Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-26T04:18:50.788 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.665042707Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-26T04:18:50.788 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.667771599Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-26T04:18:50.788 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.670435286Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-26T04:18:50.788 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.67078282Z level=info msg="Executing migration" id="Move region to single row" 2024-03-26T04:18:50.788 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.673593533Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-26T04:18:50.788 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.674020145Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-26T04:18:50.788 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.676254697Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-26T04:18:50.788 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.679035602Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-26T04:18:50.788 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.67949998Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-26T04:18:50.788 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.68224829Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-26T04:18:50.789 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.684960483Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-26T04:18:50.789 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.687569556Z level=info msg="Executing migration" id="create test_data table" 2024-03-26T04:18:50.789 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.688003483Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-26T04:18:50.789 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.690708342Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-26T04:18:50.789 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.693243756Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-26T04:18:50.789 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.695985917Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-26T04:18:50.789 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.696474045Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-26T04:18:50.789 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.699190511Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-26T04:18:50.789 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.701763161Z level=info msg="Executing migration" id="create team table" 2024-03-26T04:18:50.789 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.70450224Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-26T04:18:50.789 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.704976897Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-26T04:18:50.789 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.70781155Z level=info msg="Executing migration" id="create team member table" 2024-03-26T04:18:50.789 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.710447077Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-26T04:18:50.789 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.713104878Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-26T04:18:50.790 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.713626026Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-26T04:18:50.790 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.716320561Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-26T04:18:50.790 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.719028593Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-26T04:18:50.790 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.719997157Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-26T04:18:50.790 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.720749414Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-26T04:18:50.790 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.721754505Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-26T04:18:50.790 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.722448248Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-26T04:18:50.790 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.723394701Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-26T04:18:50.790 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.723956364Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-26T04:18:50.790 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.724758773Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-26T04:18:50.790 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.725681508Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-26T04:18:50.790 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.726390833Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-26T04:18:50.790 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.727215626Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-26T04:18:50.791 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.728170826Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-26T04:18:50.791 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.728878612Z level=info msg="Executing migration" id="create tag table" 2024-03-26T04:18:50.791 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.729758115Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-26T04:18:50.791 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.730514956Z level=info msg="Executing migration" id="create login attempt table" 2024-03-26T04:18:50.791 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.731320661Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-26T04:18:50.791 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.73222579Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-26T04:18:50.791 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.733071418Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-26T04:18:50.791 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.735190978Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-26T04:18:50.792 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.736028799Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-26T04:18:50.792 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.736894713Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-26T04:18:50.792 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.737758544Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-26T04:18:50.792 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.738624438Z level=info msg="Executing migration" id="create user auth table" 2024-03-26T04:18:50.792 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.739499154Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-26T04:18:50.792 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.740416079Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-26T04:18:50.792 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.741176361Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-26T04:18:50.793 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.742009227Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-26T04:18:50.793 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.74285666Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-26T04:18:50.793 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.743731626Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-26T04:18:50.793 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.744574171Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-26T04:18:50.793 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.745542905Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-26T04:18:50.793 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.746494816Z level=info msg="Executing migration" id="create server_lock table" 2024-03-26T04:18:50.793 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.747510073Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-26T04:18:50.793 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.748220377Z level=info msg="Executing migration" id="create user auth token table" 2024-03-26T04:18:50.793 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.749182695Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-26T04:18:50.793 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.75003896Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-26T04:18:50.793 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.752796545Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-26T04:18:50.794 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.755357788Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-26T04:18:50.794 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.758281741Z level=info msg="Executing migration" id="create cache_data table" 2024-03-26T04:18:50.794 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.760779718Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-26T04:18:50.794 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.763684247Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-26T04:18:50.794 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.766481892Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-26T04:18:50.794 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.769200598Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-26T04:18:50.794 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.771955044Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-26T04:18:50.794 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.774751733Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-26T04:18:50.794 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.777453725Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-26T04:18:50.794 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.780102839Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-26T04:18:51.042 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.782876178Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-26T04:18:51.042 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.785742484Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-26T04:18:51.042 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.788495765Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-26T04:18:51.042 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.791285816Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-26T04:18:51.042 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.793916577Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-26T04:18:51.042 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.796723985Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-26T04:18:51.043 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.799444147Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-26T04:18:51.043 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.799868991Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-26T04:18:51.043 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.800226537Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-26T04:18:51.043 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.800640851Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-26T04:18:51.043 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.801037465Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-26T04:18:51.043 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.801462093Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-26T04:18:51.043 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.801750847Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-26T04:18:51.043 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.802137458Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-26T04:18:51.043 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.802552876Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-26T04:18:51.043 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.802931983Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-26T04:18:51.044 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.803364813Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-26T04:18:51.044 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.804127235Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-26T04:18:51.044 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.804552775Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-26T04:18:51.044 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.804923479Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-26T04:18:51.044 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.808294393Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-26T04:18:51.044 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.811630676Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-26T04:18:51.044 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.812070917Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-26T04:18:51.044 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.812502257Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-26T04:18:51.044 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.813355213Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-26T04:18:51.044 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.813763771Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-26T04:18:51.044 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.814200437Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-26T04:18:51.045 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.814618299Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-26T04:18:51.045 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.815018725Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-26T04:18:51.045 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.815314573Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-26T04:18:51.045 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.816102357Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-26T04:18:51.045 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.816890899Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-26T04:18:51.045 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.81768199Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-26T04:18:51.045 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.818083228Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-26T04:18:51.045 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.818532809Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-26T04:18:51.047 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.819347395Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-26T04:18:51.047 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.820153842Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-26T04:18:51.047 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.820601531Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-26T04:18:51.047 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.82146792Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-26T04:18:51.047 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.822278545Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-26T04:18:51.047 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.822628506Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-26T04:18:51.047 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.823131275Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-26T04:18:51.047 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.825907525Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-26T04:18:51.048 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.826388421Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-26T04:18:51.048 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.826691275Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-26T04:18:51.048 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.827575161Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-26T04:18:51.048 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.830327268Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-26T04:18:51.048 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.831191092Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-26T04:18:51.048 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.832009512Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-26T04:18:51.048 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.832845259Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-26T04:18:51.048 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.833188432Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-26T04:18:51.048 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.833600675Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-26T04:18:51.048 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.83446559Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-26T04:18:51.048 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.834792929Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-26T04:18:51.048 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.835689737Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-26T04:18:51.048 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.838313974Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-26T04:18:51.049 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.84126825Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-26T04:18:51.049 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.843833303Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-26T04:18:51.049 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.846592573Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-26T04:18:51.049 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.849120597Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-26T04:18:51.049 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.851773788Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-26T04:18:51.049 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.854489893Z level=info msg="Executing migration" id="create alert_image table" 2024-03-26T04:18:51.049 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.857221955Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-26T04:18:51.049 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.859805503Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-26T04:18:51.049 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.862490897Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-26T04:18:51.049 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.863174146Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-26T04:18:51.049 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.865882778Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-26T04:18:51.049 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.866089451Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-26T04:18:51.049 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.868509249Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-26T04:18:51.049 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.871123264Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-26T04:18:51.050 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.873655322Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-26T04:18:51.050 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.873803236Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-26T04:18:51.050 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.873914961Z level=info msg="alerts found to migrate" alerts=0 2024-03-26T04:18:51.050 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.87638484Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-26T04:18:51.050 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.879075563Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-26T04:18:51.050 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.881678121Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-26T04:18:51.050 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.884326923Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-26T04:18:51.050 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.887073283Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-26T04:18:51.051 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.889758126Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-26T04:18:51.051 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.89061414Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-26T04:18:51.051 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.891364197Z level=info msg="Executing migration" id="create data_keys table" 2024-03-26T04:18:51.051 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.892338415Z level=info msg="Executing migration" id="create secrets table" 2024-03-26T04:18:51.051 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.89304379Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-26T04:18:51.051 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.897205205Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-26T04:18:51.051 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.898095782Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-26T04:18:51.051 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.898867809Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-26T04:18:51.051 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.903075702Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-26T04:18:51.051 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.907279071Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-26T04:18:51.052 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.908066586Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-26T04:18:51.052 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.908972473Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-26T04:18:51.052 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.909691166Z level=info msg="Executing migration" id="create permission table" 2024-03-26T04:18:51.052 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.910617033Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-26T04:18:51.052 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.911468622Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-26T04:18:51.052 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.912272515Z level=info msg="Executing migration" id="create role table" 2024-03-26T04:18:51.052 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.913014547Z level=info msg="Executing migration" id="add column display_name" 2024-03-26T04:18:51.052 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.913939744Z level=info msg="Executing migration" id="add column group_name" 2024-03-26T04:18:51.052 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.914970221Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-26T04:18:51.053 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.916009378Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-26T04:18:51.053 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.916750211Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-26T04:18:51.053 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.917752422Z level=info msg="Executing migration" id="create team role table" 2024-03-26T04:18:51.053 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.918521285Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-26T04:18:51.053 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.91949981Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-26T04:18:51.053 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.920223424Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-26T04:18:51.053 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.921193809Z level=info msg="Executing migration" id="create user role table" 2024-03-26T04:18:51.054 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.921943449Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-26T04:18:51.054 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.922879074Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-26T04:18:51.054 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.923620644Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-26T04:18:51.054 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.924615593Z level=info msg="Executing migration" id="create builtin role table" 2024-03-26T04:18:51.054 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.925445881Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-26T04:18:51.054 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.926185565Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-26T04:18:51.055 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.927052805Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-26T04:18:51.055 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.928024036Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-26T04:18:51.055 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.928520324Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-26T04:18:51.055 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.929252923Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-26T04:18:51.055 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.929693891Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-26T04:18:51.055 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.930121498Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-26T04:18:51.055 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.930536823Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-26T04:18:51.056 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.930993241Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-26T04:18:51.056 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.931954055Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-26T04:18:51.056 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.932400135Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-26T04:18:51.056 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.932852982Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-26T04:18:51.056 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.933135481Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-26T04:18:51.057 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.933569541Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-26T04:18:51.057 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.933977308Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-26T04:18:51.057 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.934352483Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-26T04:18:51.057 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.934659383Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-26T04:18:51.057 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.934976951Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-26T04:18:51.057 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.935372197Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-26T04:18:51.057 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.935802192Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-26T04:18:51.058 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.936790087Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-26T04:18:51.058 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.937089205Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-26T04:18:51.058 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.937525926Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-26T04:18:51.058 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.937947315Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-26T04:18:51.058 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.938392714Z level=info msg="Executing migration" id="add correlation config column" 2024-03-26T04:18:51.058 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.939378436Z level=info msg="Executing migration" id="create entity_events table" 2024-03-26T04:18:51.059 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.939856776Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-26T04:18:51.059 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.942666893Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-26T04:18:51.059 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.942773249Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-26T04:18:51.059 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.943072409Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-26T04:18:51.059 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.943161886Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-26T04:18:51.059 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.943436204Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-26T04:18:51.059 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.943813482Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-26T04:18:51.060 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.944196899Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-26T04:18:51.060 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.94462333Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-26T04:18:51.060 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.945061712Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-26T04:18:51.060 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.945496707Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-26T04:18:51.061 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.945923491Z level=info msg="Executing migration" id="Drop public config table" 2024-03-26T04:18:51.061 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.94631318Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-26T04:18:51.061 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.946722974Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-26T04:18:51.061 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.947116342Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-26T04:18:51.061 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.947584651Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-26T04:18:51.061 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.949921735Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-26T04:18:51.061 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.952932963Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-26T04:18:51.061 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.955645218Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-26T04:18:51.062 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.958335161Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-26T04:18:51.062 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.961163648Z level=info msg="Executing migration" id="add share column" 2024-03-26T04:18:51.062 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.963816979Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-26T04:18:51.062 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.96627689Z level=info msg="Executing migration" id="create file table" 2024-03-26T04:18:51.062 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.968966345Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-26T04:18:51.062 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.971750615Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-26T04:18:51.062 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.974445143Z level=info msg="Executing migration" id="create file_meta table" 2024-03-26T04:18:51.063 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.977042549Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-26T04:18:51.063 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.979788253Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-26T04:18:51.063 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.982252118Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-26T04:18:51.063 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.985097818Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-26T04:18:51.063 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.987747572Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-26T04:18:51.063 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.99021165Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-26T04:18:51.063 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.992908477Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-26T04:18:51.064 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.995718266Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-26T04:18:51.064 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.996423777Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-26T04:18:51.064 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.997321061Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-26T04:18:51.064 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.998021929Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-26T04:18:51.064 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:50 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:50.998979077Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-26T04:18:51.064 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:51.000182678Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-26T04:18:51.064 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:51.00122108Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-26T04:18:51.064 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:51.002020441Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-26T04:18:51.065 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:51.011776275Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-26T04:18:51.065 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:51.012542109Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-26T04:18:51.065 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:51.013348232Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-26T04:18:51.065 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:51.016905774Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-26T04:18:51.066 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:51.017695579Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-26T04:18:51.066 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:51.018660922Z level=info msg="Executing migration" id="create folder table" 2024-03-26T04:18:51.066 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:51.019380082Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-26T04:18:51.066 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:51.020180731Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-26T04:18:51.066 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:51.020985358Z level=info msg="Executing migration" id="Update folder title length" 2024-03-26T04:18:51.066 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:51.021819111Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-26T04:18:51.066 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=migrator t=2024-03-26T04:18:51.022297169Z level=info msg="migrations completed" performed=485 skipped=0 duration=658.870098ms 2024-03-26T04:18:51.066 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=sqlstore t=2024-03-26T04:18:51.022694374Z level=info msg="Created default organization" 2024-03-26T04:18:51.330 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=plugin.loader t=2024-03-26T04:18:51.080519264Z level=info msg="Plugin registered" pluginID=input 2024-03-26T04:18:51.331 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=plugin.loader t=2024-03-26T04:18:51.108775782Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-03-26T04:18:51.331 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=plugin.loader t=2024-03-26T04:18:51.108822949Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-03-26T04:18:51.331 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=secrets t=2024-03-26T04:18:51.108856791Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-26T04:18:51.331 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=query_data t=2024-03-26T04:18:51.110958369Z level=info msg="Query Service initialization" 2024-03-26T04:18:51.331 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=live.push_http t=2024-03-26T04:18:51.113343354Z level=info msg="Live Push Gateway initialization" 2024-03-26T04:18:51.604 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:51.604 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:51.604 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:51.604 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:51.604 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-mon[33745]: Deploying daemon node-exporter.a on smithi016 2024-03-26T04:18:51.604 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-03-26T04:18:51.604 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-03-26T04:18:51.604 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-03-26T04:18:51.605 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-03-26T04:18:51.605 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-03-26T04:18:51.605 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-03-26T04:18:51.605 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:51.605 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:18:51.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:51 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:51.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:51 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:51.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:51 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:51.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:51 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:51.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:51 smithi016 ceph-mon[37215]: Deploying daemon node-exporter.a on smithi016 2024-03-26T04:18:51.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:51 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-03-26T04:18:51.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:51 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-03-26T04:18:51.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:51 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-03-26T04:18:51.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:51 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-03-26T04:18:51.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:51 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-03-26T04:18:51.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:51 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-03-26T04:18:51.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:51 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:51.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:51 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:18:51.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:51 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:51.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:51 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:51.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:51 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:51.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:51 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:51.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:51 smithi016 ceph-mon[29101]: Deploying daemon node-exporter.a on smithi016 2024-03-26T04:18:51.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:51 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-03-26T04:18:51.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:51 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-03-26T04:18:51.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:51 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2024-03-26T04:18:51.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:51 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-03-26T04:18:51.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:51 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]: dispatch 2024-03-26T04:18:51.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:51 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]: dispatch 2024-03-26T04:18:51.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:51 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:51.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:51 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:18:51.665 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:51 smithi016 bash[72629]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-26T04:18:51.978 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=infra.usagestats.collector t=2024-03-26T04:18:51.738611655Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-26T04:18:51.978 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=provisioning.datasources t=2024-03-26T04:18:51.739705271Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-26T04:18:51.978 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=provisioning.datasources t=2024-03-26T04:18:51.75098939Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-26T04:18:51.978 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=provisioning.alerting t=2024-03-26T04:18:51.762533158Z level=info msg="starting to provision alerting" 2024-03-26T04:18:51.978 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=provisioning.alerting t=2024-03-26T04:18:51.76257612Z level=info msg="finished to provision alerting" 2024-03-26T04:18:51.978 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=grafanaStorageLogger t=2024-03-26T04:18:51.762755442Z level=info msg="storage starting" 2024-03-26T04:18:51.978 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=ngalert.state.manager t=2024-03-26T04:18:51.762947977Z level=info msg="Warming state cache for startup" 2024-03-26T04:18:51.978 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=http.server t=2024-03-26T04:18:51.764694162Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-26T04:18:51.978 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=ngalert.state.manager t=2024-03-26T04:18:51.767352163Z level=info msg="State cache has been initialized" states=0 duration=4.403055ms 2024-03-26T04:18:51.979 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=ticker t=2024-03-26T04:18:51.767401929Z level=info msg=starting first_tick=2024-03-26T04:19:00Z 2024-03-26T04:18:51.979 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=ngalert.multiorg.alertmanager t=2024-03-26T04:18:51.767416694Z level=info msg="starting MultiOrg Alertmanager" 2024-03-26T04:18:51.979 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=sqlstore.transactions t=2024-03-26T04:18:51.773425833Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-26T04:18:51.979 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=sqlstore.transactions t=2024-03-26T04:18:51.777732349Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-26T04:18:51.979 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=sqlstore.transactions t=2024-03-26T04:18:51.784306469Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-03-26T04:18:51.979 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=sqlstore.transactions t=2024-03-26T04:18:51.795986544Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=2 code="database is locked" 2024-03-26T04:18:51.979 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=sqlstore.transactions t=2024-03-26T04:18:51.806321075Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=3 code="database is locked" 2024-03-26T04:18:52.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:18:51 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:18:52.360 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:51 smithi016 bash[72629]: Getting image source signatures 2024-03-26T04:18:52.360 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:51 smithi016 bash[72629]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-26T04:18:52.360 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:51 smithi016 bash[72629]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-26T04:18:52.360 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:51 smithi016 bash[72629]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-26T04:18:52.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:52 smithi016 ceph-mon[29101]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:18:52.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:52 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-03-26T04:18:52.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:52 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-03-26T04:18:52.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:52 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2024-03-26T04:18:52.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:52 smithi016 ceph-mon[29101]: osdmap e71: 8 total, 8 up, 8 in 2024-03-26T04:18:52.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:52 smithi016 ceph-mon[37215]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:18:52.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:52 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-03-26T04:18:52.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:52 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-03-26T04:18:52.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:52 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2024-03-26T04:18:52.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:52 smithi016 ceph-mon[37215]: osdmap e71: 8 total, 8 up, 8 in 2024-03-26T04:18:52.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:52 smithi183 ceph-mon[33745]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:18:52.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:52 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2024-03-26T04:18:52.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:52 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.d", "id": [1, 5]}]': finished 2024-03-26T04:18:52.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:52 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [1, 5]}]': finished 2024-03-26T04:18:52.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:52 smithi183 ceph-mon[33745]: osdmap e71: 8 total, 8 up, 8 in 2024-03-26T04:18:53.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:53 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:18:53.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:53 smithi016 ceph-mon[29101]: osdmap e72: 8 total, 8 up, 8 in 2024-03-26T04:18:53.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:53 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:18:53.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:53 smithi016 ceph-mon[37215]: osdmap e72: 8 total, 8 up, 8 in 2024-03-26T04:18:53.663 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:53 smithi016 bash[72629]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-26T04:18:53.663 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:53 smithi016 bash[72629]: Writing manifest to image destination 2024-03-26T04:18:53.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:53 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:18:53.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:53 smithi183 ceph-mon[33745]: osdmap e72: 8 total, 8 up, 8 in 2024-03-26T04:18:54.110 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:53 smithi016 podman[72629]: 2024-03-26 04:18:53.750704704 +0000 UTC m=+2.316855713 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-26T04:18:54.110 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:53 smithi016 podman[72629]: 2024-03-26 04:18:53.840968802 +0000 UTC m=+2.407119812 container create 115b83f6ff7c5286859de9598beeb2047f7f14d4ca13f8c5db5c4f394f28ea0b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T04:18:54.110 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 podman[72629]: 2024-03-26 04:18:54.110246865 +0000 UTC m=+2.676397877 container init 115b83f6ff7c5286859de9598beeb2047f7f14d4ca13f8c5db5c4f394f28ea0b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T04:18:54.366 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 podman[72629]: 2024-03-26 04:18:54.113305439 +0000 UTC m=+2.679456455 container start 115b83f6ff7c5286859de9598beeb2047f7f14d4ca13f8c5db5c4f394f28ea0b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T04:18:54.367 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.117Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-26T04:18:54.367 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.117Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-26T04:18:54.367 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.118Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-26T04:18:54.367 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.118Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-26T04:18:54.367 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.118Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-26T04:18:54.367 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.118Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-26T04:18:54.367 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-26T04:18:54.367 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=node_exporter.go:117 level=info collector=arp 2024-03-26T04:18:54.367 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-26T04:18:54.367 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-26T04:18:54.367 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-26T04:18:54.367 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-26T04:18:54.367 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-26T04:18:54.368 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-26T04:18:54.368 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-26T04:18:54.368 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-26T04:18:54.368 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=node_exporter.go:117 level=info collector=edac 2024-03-26T04:18:54.368 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-26T04:18:54.368 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-26T04:18:54.368 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-26T04:18:54.369 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-26T04:18:54.369 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-26T04:18:54.369 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-26T04:18:54.369 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-26T04:18:54.369 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-26T04:18:54.369 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-26T04:18:54.369 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-26T04:18:54.369 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-26T04:18:54.369 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-26T04:18:54.369 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-26T04:18:54.369 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-26T04:18:54.369 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-26T04:18:54.369 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-26T04:18:54.369 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=node_exporter.go:117 level=info collector=os 2024-03-26T04:18:54.369 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-26T04:18:54.370 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-26T04:18:54.370 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-26T04:18:54.370 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-26T04:18:54.370 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-26T04:18:54.370 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-26T04:18:54.370 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-26T04:18:54.370 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=node_exporter.go:117 level=info collector=stat 2024-03-26T04:18:54.370 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-26T04:18:54.370 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-26T04:18:54.370 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-26T04:18:54.370 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=node_exporter.go:117 level=info collector=time 2024-03-26T04:18:54.370 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-26T04:18:54.370 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=node_exporter.go:117 level=info collector=uname 2024-03-26T04:18:54.371 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-26T04:18:54.371 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-26T04:18:54.371 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-26T04:18:54.371 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-26T04:18:54.371 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a[72681]: ts=2024-03-26T04:18:54.119Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-26T04:18:54.371 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 bash[72629]: 115b83f6ff7c5286859de9598beeb2047f7f14d4ca13f8c5db5c4f394f28ea0b 2024-03-26T04:18:54.371 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:18:54 smithi016 systemd[1]: Started Ceph node-exporter.a for d7d59826-eb26-11ee-95d0-87774f69a715. 2024-03-26T04:18:54.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-mon[29101]: pgmap v36: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:18:54.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-mon[29101]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-03-26T04:18:54.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:54.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:54.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:54.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-mon[37215]: pgmap v36: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:18:54.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-mon[37215]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-03-26T04:18:54.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:54.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:54.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:54 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:54.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:54 smithi183 ceph-mon[33745]: pgmap v36: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:18:54.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:54 smithi183 ceph-mon[33745]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-03-26T04:18:54.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:54 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:54.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:54 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:54.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:54 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:55.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:55 smithi016 ceph-mon[29101]: Deploying daemon node-exporter.b on smithi183 2024-03-26T04:18:55.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:55 smithi016 ceph-mon[37215]: Deploying daemon node-exporter.b on smithi183 2024-03-26T04:18:55.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:55 smithi183 ceph-mon[33745]: Deploying daemon node-exporter.b on smithi183 2024-03-26T04:18:55.792 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:55 smithi183 bash[64173]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-26T04:18:56.292 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:55 smithi183 bash[64173]: Getting image source signatures 2024-03-26T04:18:56.292 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:55 smithi183 bash[64173]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-26T04:18:56.292 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:55 smithi183 bash[64173]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-26T04:18:56.292 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:55 smithi183 bash[64173]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-26T04:18:56.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:56 smithi016 ceph-mon[29101]: pgmap v37: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:18:56.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:56 smithi016 ceph-mon[37215]: pgmap v37: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:18:56.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:56 smithi183 ceph-mon[33745]: pgmap v37: 132 pgs: 3 peering, 129 active+clean; 582 KiB data, 238 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:18:57.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:57 smithi183 ceph-mon[33745]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-03-26T04:18:57.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:57 smithi183 ceph-mon[33745]: Cluster is now healthy 2024-03-26T04:18:57.542 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:57 smithi183 bash[64173]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-26T04:18:57.542 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:57 smithi183 bash[64173]: Writing manifest to image destination 2024-03-26T04:18:57.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:57 smithi016 ceph-mon[29101]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-03-26T04:18:57.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:57 smithi016 ceph-mon[29101]: Cluster is now healthy 2024-03-26T04:18:57.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:57 smithi016 ceph-mon[37215]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-03-26T04:18:57.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:57 smithi016 ceph-mon[37215]: Cluster is now healthy 2024-03-26T04:18:58.042 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:57 smithi183 podman[64173]: 2024-03-26 04:18:57.679069973 +0000 UTC m=+2.262139126 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-26T04:18:58.042 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:57 smithi183 podman[64173]: 2024-03-26 04:18:57.81952885 +0000 UTC m=+2.402597988 container create 4a3fac17f30bb222a075379ece2658d5215d1d3b44b84bcfb971924fd227c033 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T04:18:58.425 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 podman[64173]: 2024-03-26 04:18:58.046541986 +0000 UTC m=+2.629611127 container init 4a3fac17f30bb222a075379ece2658d5215d1d3b44b84bcfb971924fd227c033 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T04:18:58.425 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 podman[64173]: 2024-03-26 04:18:58.04901202 +0000 UTC m=+2.632081159 container start 4a3fac17f30bb222a075379ece2658d5215d1d3b44b84bcfb971924fd227c033 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T04:18:58.425 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.052Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-26T04:18:58.425 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.052Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-26T04:18:58.425 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.053Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-26T04:18:58.425 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.053Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-26T04:18:58.425 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.053Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-26T04:18:58.426 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.053Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-26T04:18:58.426 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.053Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-26T04:18:58.426 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.053Z caller=node_exporter.go:117 level=info collector=arp 2024-03-26T04:18:58.426 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.053Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-26T04:18:58.426 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.053Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-26T04:18:58.426 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.053Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-26T04:18:58.426 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.053Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-26T04:18:58.426 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.053Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-26T04:18:58.426 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.053Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-26T04:18:58.427 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.053Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-26T04:18:58.427 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.053Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-26T04:18:58.427 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.053Z caller=node_exporter.go:117 level=info collector=edac 2024-03-26T04:18:58.427 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.053Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-26T04:18:58.427 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.053Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-26T04:18:58.427 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.053Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-26T04:18:58.427 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.053Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-26T04:18:58.427 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.053Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-26T04:18:58.427 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.053Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-26T04:18:58.427 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.053Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-26T04:18:58.427 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.053Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-26T04:18:58.427 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.053Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-26T04:18:58.428 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.053Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-26T04:18:58.428 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.053Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-26T04:18:58.428 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.053Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-26T04:18:58.428 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.053Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-26T04:18:58.428 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.053Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-26T04:18:58.428 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.053Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-26T04:18:58.428 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.053Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-26T04:18:58.428 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.053Z caller=node_exporter.go:117 level=info collector=os 2024-03-26T04:18:58.428 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.053Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-26T04:18:58.428 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.053Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-26T04:18:58.428 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.053Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-26T04:18:58.428 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.053Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-26T04:18:58.428 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.053Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-26T04:18:58.428 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.053Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-26T04:18:58.429 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.054Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-26T04:18:58.429 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.054Z caller=node_exporter.go:117 level=info collector=stat 2024-03-26T04:18:58.429 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.054Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-26T04:18:58.429 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.054Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-26T04:18:58.429 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.054Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-26T04:18:58.429 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.054Z caller=node_exporter.go:117 level=info collector=time 2024-03-26T04:18:58.429 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.054Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-26T04:18:58.429 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.054Z caller=node_exporter.go:117 level=info collector=uname 2024-03-26T04:18:58.429 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.054Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-26T04:18:58.429 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.054Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-26T04:18:58.429 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.054Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-26T04:18:58.429 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.054Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-26T04:18:58.429 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b[64223]: ts=2024-03-26T04:18:58.054Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-26T04:18:58.430 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 bash[64173]: 4a3fac17f30bb222a075379ece2658d5215d1d3b44b84bcfb971924fd227c033 2024-03-26T04:18:58.430 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:18:58 smithi183 systemd[1]: Started Ceph node-exporter.b for d7d59826-eb26-11ee-95d0-87774f69a715. 2024-03-26T04:18:58.745 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:58 smithi016 ceph-mon[37215]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41 B/s, 3 objects/s recovering 2024-03-26T04:18:58.746 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:58 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:58.746 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:58 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:58.746 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:58 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:58.746 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:58 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:58.746 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:58 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:58.746 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:18:58 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:18:58.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:58 smithi016 ceph-mon[29101]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41 B/s, 3 objects/s recovering 2024-03-26T04:18:58.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:58 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:58.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:58 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:58.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:58 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:58.747 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:58 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:58.747 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:58 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:58.747 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:18:58 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:18:58.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-mon[33745]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41 B/s, 3 objects/s recovering 2024-03-26T04:18:58.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:58.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:58.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:58.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:58.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:18:58.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:18:58 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:18:59.159 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:18:58 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:18:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:19:00.694 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:00 smithi016 ceph-mon[37215]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41 B/s, 3 objects/s recovering 2024-03-26T04:19:00.694 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:00 smithi016 ceph-mon[29101]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41 B/s, 3 objects/s recovering 2024-03-26T04:19:00.746 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:00 smithi183 ceph-mon[33745]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 41 B/s, 3 objects/s recovering 2024-03-26T04:19:02.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:19:01 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:19:02.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:02 smithi183 ceph-mon[33745]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 34 B/s, 2 objects/s recovering 2024-03-26T04:19:02.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:02 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:02.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:02 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:02.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:02 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:02.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:02 smithi016 ceph-mon[29101]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 34 B/s, 2 objects/s recovering 2024-03-26T04:19:02.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:02 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:02.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:02 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:02.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:02 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:02.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:02 smithi016 ceph-mon[37215]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 34 B/s, 2 objects/s recovering 2024-03-26T04:19:02.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:02 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:02.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:02 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:02.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:02 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:03.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:03 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:19:03.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:03 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:03.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:03 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:03.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:03 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:19:03.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:03 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:03.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:03 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:03.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:03 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:19:03.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:03 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:03.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:03 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:04.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:04 smithi183 ceph-mon[33745]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 965 B/s rd, 0 op/s; 31 B/s, 2 objects/s recovering 2024-03-26T04:19:04.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:04 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:19:04.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:04 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:19:04.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:04 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:04.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:04 smithi183 ceph-mon[33745]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-26T04:19:04.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:04 smithi183 ceph-mon[33745]: Reconfiguring daemon alertmanager.a on smithi016 2024-03-26T04:19:04.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:04 smithi016 ceph-mon[37215]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 965 B/s rd, 0 op/s; 31 B/s, 2 objects/s recovering 2024-03-26T04:19:04.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:04 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:19:04.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:04 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:19:04.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:04 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:04.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:04 smithi016 ceph-mon[37215]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-26T04:19:04.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:04 smithi016 ceph-mon[37215]: Reconfiguring daemon alertmanager.a on smithi016 2024-03-26T04:19:04.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:04 smithi016 ceph-mon[29101]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 965 B/s rd, 0 op/s; 31 B/s, 2 objects/s recovering 2024-03-26T04:19:04.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:04 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:19:04.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:04 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:19:04.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:04 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:04.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:04 smithi016 ceph-mon[29101]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-26T04:19:04.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:04 smithi016 ceph-mon[29101]: Reconfiguring daemon alertmanager.a on smithi016 2024-03-26T04:19:06.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:06 smithi183 ceph-mon[33745]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 27 B/s, 2 objects/s recovering 2024-03-26T04:19:06.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:06 smithi016 ceph-mon[37215]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 27 B/s, 2 objects/s recovering 2024-03-26T04:19:06.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:06 smithi016 ceph-mon[29101]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 27 B/s, 2 objects/s recovering 2024-03-26T04:19:07.290 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:07 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:07.290 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:07 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:19:07.290 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:19:07 smithi016 systemd[1]: Stopping Ceph alertmanager.a for d7d59826-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:19:07.291 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:07 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:07.291 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:07 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:19:07.339 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:07 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:07.339 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:07 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:19:07.623 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:19:07 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-alertmanager-a[72105]: ts=2024-03-26T04:19:07.289Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-26T04:19:07.624 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:19:07 smithi016 podman[73410]: 2024-03-26 04:19:07.32010677 +0000 UTC m=+0.153055163 container died 7cd00f19dd8a9e5b92136fa7669422dd9c52af1d669b75c82e4495a73f9c033e (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:19:07.912 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:19:07 smithi016 podman[73410]: 2024-03-26 04:19:07.832309721 +0000 UTC m=+0.665258121 container cleanup 7cd00f19dd8a9e5b92136fa7669422dd9c52af1d669b75c82e4495a73f9c033e (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:19:07.912 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:19:07 smithi016 bash[73410]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715-alertmanager-a 2024-03-26T04:19:08.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:08 smithi016 ceph-mon[37215]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 2 objects/s recovering 2024-03-26T04:19:08.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:08 smithi016 ceph-mon[29101]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 2 objects/s recovering 2024-03-26T04:19:08.413 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:19:08 smithi016 podman[73423]: 2024-03-26 04:19:08.075517202 +0000 UTC m=+0.757502261 container remove 7cd00f19dd8a9e5b92136fa7669422dd9c52af1d669b75c82e4495a73f9c033e (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:19:08.414 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:19:08 smithi016 podman[73423]: 2024-03-26 04:19:08.176336135 +0000 UTC m=+0.858321196 volume remove 37b1d63f990fb417648404732f5ec085c0fad75e41dcc7dadf50285a1152b5ee 2024-03-26T04:19:08.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:08 smithi183 ceph-mon[33745]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 2 objects/s recovering 2024-03-26T04:19:08.746 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:19:08 smithi016 systemd[1]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-26T04:19:08.746 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:19:08 smithi016 systemd[1]: Stopped Ceph alertmanager.a for d7d59826-eb26-11ee-95d0-87774f69a715. 2024-03-26T04:19:08.746 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:19:08 smithi016 systemd[1]: Starting Ceph alertmanager.a for d7d59826-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:19:08.746 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:19:08 smithi016 podman[73514]: 2024-03-26 04:19:08.547601226 +0000 UTC m=+0.017927242 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-26T04:19:08.746 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:19:08 smithi016 podman[73514]: 2024-03-26 04:19:08.660823197 +0000 UTC m=+0.131149198 volume create 38d4389a60992805c767df881a5fdd32ba1f98f9f3dd0818c1b010de25eb6c1e 2024-03-26T04:19:09.098 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:19:08 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:19:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:19:09.098 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:19:08 smithi016 podman[73514]: 2024-03-26 04:19:08.761356899 +0000 UTC m=+0.231682900 container create 6550a87ba310b28d3bb3e5d96eb439b32ddc74613365f608aeb975cb0873a1a4 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:19:09.412 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:19:09 smithi016 podman[73514]: 2024-03-26 04:19:09.098461096 +0000 UTC m=+0.568787100 container init 6550a87ba310b28d3bb3e5d96eb439b32ddc74613365f608aeb975cb0873a1a4 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:19:09.412 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:19:09 smithi016 podman[73514]: 2024-03-26 04:19:09.102432666 +0000 UTC m=+0.572758675 container start 6550a87ba310b28d3bb3e5d96eb439b32ddc74613365f608aeb975cb0873a1a4 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:19:09.412 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:19:09 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-alertmanager-a[73529]: ts=2024-03-26T04:19:09.116Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-26T04:19:09.413 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:19:09 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-alertmanager-a[73529]: ts=2024-03-26T04:19:09.116Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-26T04:19:09.413 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:19:09 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-alertmanager-a[73529]: ts=2024-03-26T04:19:09.117Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.16 port=9094 2024-03-26T04:19:09.413 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:19:09 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-alertmanager-a[73529]: ts=2024-03-26T04:19:09.118Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-26T04:19:09.413 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:19:09 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-alertmanager-a[73529]: ts=2024-03-26T04:19:09.143Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T04:19:09.413 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:19:09 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-alertmanager-a[73529]: ts=2024-03-26T04:19:09.143Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-26T04:19:09.413 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:19:09 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-alertmanager-a[73529]: ts=2024-03-26T04:19:09.146Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-26T04:19:09.413 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:19:09 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-alertmanager-a[73529]: ts=2024-03-26T04:19:09.146Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-26T04:19:09.413 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:19:09 smithi016 bash[73514]: 6550a87ba310b28d3bb3e5d96eb439b32ddc74613365f608aeb975cb0873a1a4 2024-03-26T04:19:09.413 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:19:09 smithi016 systemd[1]: Started Ceph alertmanager.a for d7d59826-eb26-11ee-95d0-87774f69a715. 2024-03-26T04:19:10.441 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:10 smithi183 ceph-mon[33745]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:19:10.442 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:10 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:10.442 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:10 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:10.442 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:10 smithi183 ceph-mon[33745]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T04:19:10.442 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:10 smithi183 ceph-mon[33745]: Reconfiguring daemon prometheus.a on smithi183 2024-03-26T04:19:10.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:10 smithi016 ceph-mon[37215]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:19:10.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:10 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:10.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:10 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:10.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:10 smithi016 ceph-mon[37215]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T04:19:10.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:10 smithi016 ceph-mon[37215]: Reconfiguring daemon prometheus.a on smithi183 2024-03-26T04:19:10.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:10 smithi016 ceph-mon[29101]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:19:10.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:10 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:10.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:10 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:10.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:10 smithi016 ceph-mon[29101]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-26T04:19:10.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:10 smithi016 ceph-mon[29101]: Reconfiguring daemon prometheus.a on smithi183 2024-03-26T04:19:11.412 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:19:11 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-alertmanager-a[73529]: ts=2024-03-26T04:19:11.118Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000057629s 2024-03-26T04:19:12.041 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:19:12 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:19:12.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:12 smithi183 ceph-mon[33745]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:19:12.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:12 smithi016 ceph-mon[37215]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:19:12.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:12 smithi016 ceph-mon[29101]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:19:12.905 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:12 smithi183 systemd[1]: Stopping Ceph prometheus.a for d7d59826-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:19:13.172 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:12 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[62004]: ts=2024-03-26T04:19:12.905Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-26T04:19:13.172 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:12 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[62004]: ts=2024-03-26T04:19:12.905Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-26T04:19:13.172 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:12 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[62004]: ts=2024-03-26T04:19:12.905Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-26T04:19:13.172 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:12 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[62004]: ts=2024-03-26T04:19:12.905Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-26T04:19:13.172 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:12 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[62004]: ts=2024-03-26T04:19:12.905Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-26T04:19:13.172 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:12 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[62004]: ts=2024-03-26T04:19:12.905Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-26T04:19:13.172 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:12 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[62004]: ts=2024-03-26T04:19:12.905Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-26T04:19:13.172 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:12 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[62004]: ts=2024-03-26T04:19:12.905Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-26T04:19:13.172 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:12 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[62004]: ts=2024-03-26T04:19:12.905Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-26T04:19:13.172 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:12 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[62004]: ts=2024-03-26T04:19:12.905Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-26T04:19:13.172 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:12 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[62004]: ts=2024-03-26T04:19:12.905Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-26T04:19:13.173 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:12 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[62004]: ts=2024-03-26T04:19:12.905Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-26T04:19:13.173 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:12 smithi183 podman[65127]: 2024-03-26 04:19:12.915404567 +0000 UTC m=+0.116847659 container died d5ec4698a6dbd63cf3e79ca8096b9773a5c7affbf918ee1a135b3a8372297a71 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:19:13.514 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:13 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:19:13.514 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:13 smithi183 podman[65127]: 2024-03-26 04:19:13.279708062 +0000 UTC m=+0.481151154 container cleanup d5ec4698a6dbd63cf3e79ca8096b9773a5c7affbf918ee1a135b3a8372297a71 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:19:13.514 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:13 smithi183 bash[65127]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a 2024-03-26T04:19:13.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:13 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:19:13.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:13 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:19:13.791 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:13 smithi183 podman[65142]: 2024-03-26 04:19:13.514381728 +0000 UTC m=+0.601400617 container remove d5ec4698a6dbd63cf3e79ca8096b9773a5c7affbf918ee1a135b3a8372297a71 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:19:13.792 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:13 smithi183 systemd[1]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-26T04:19:13.792 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:13 smithi183 systemd[1]: Stopped Ceph prometheus.a for d7d59826-eb26-11ee-95d0-87774f69a715. 2024-03-26T04:19:13.792 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:13 smithi183 systemd[1]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715@prometheus.a.service: Consumed 1.044s CPU time. 2024-03-26T04:19:13.792 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:13 smithi183 systemd[1]: Starting Ceph prometheus.a for d7d59826-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:19:14.113 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:13 smithi183 podman[65237]: 2024-03-26 04:19:13.754284623 +0000 UTC m=+0.019323908 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-26T04:19:14.114 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:13 smithi183 podman[65237]: 2024-03-26 04:19:13.882061534 +0000 UTC m=+0.147100847 container create 12fe18d01c61267dee1c28e0932f6235cdadea946603d73e6fcfbee503e8c905 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:19:14.114 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:14 smithi183 podman[65237]: 2024-03-26 04:19:14.114366507 +0000 UTC m=+0.379405775 container init 12fe18d01c61267dee1c28e0932f6235cdadea946603d73e6fcfbee503e8c905 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:19:14.517 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:14 smithi016 ceph-mon[37215]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:19:14.517 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:14 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:14.517 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:14 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:14.518 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:14 smithi016 ceph-mon[29101]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:19:14.518 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:14 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:14.518 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:14 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:14.518 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:19:14 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: [26/Mar/2024:04:19:14] ENGINE Bus STOPPING 2024-03-26T04:19:14.518 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:19:14 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: [26/Mar/2024:04:19:14] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T04:19:14.518 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:19:14 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: [26/Mar/2024:04:19:14] ENGINE Bus STOPPED 2024-03-26T04:19:14.518 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:19:14 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: [26/Mar/2024:04:19:14] ENGINE Bus STARTING 2024-03-26T04:19:14.518 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:19:14 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: [26/Mar/2024:04:19:14] ENGINE Serving on http://:::9283 2024-03-26T04:19:14.518 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:19:14 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: [26/Mar/2024:04:19:14] ENGINE Bus STARTED 2024-03-26T04:19:14.518 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:19:14 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: [26/Mar/2024:04:19:14] ENGINE Bus STOPPING 2024-03-26T04:19:14.519 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:19:14 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: [26/Mar/2024:04:19:14] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T04:19:14.519 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:19:14 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: [26/Mar/2024:04:19:14] ENGINE Bus STOPPED 2024-03-26T04:19:14.519 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:19:14 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: [26/Mar/2024:04:19:14] ENGINE Bus STARTING 2024-03-26T04:19:14.519 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:19:14 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: [26/Mar/2024:04:19:14] ENGINE Serving on http://:::9283 2024-03-26T04:19:14.519 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:19:14 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: [26/Mar/2024:04:19:14] ENGINE Bus STARTED 2024-03-26T04:19:14.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:14 smithi183 ceph-mon[33745]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:19:14.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:14 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:14.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:14 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:14.542 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:14 smithi183 podman[65237]: 2024-03-26 04:19:14.116864902 +0000 UTC m=+0.381904164 container start 12fe18d01c61267dee1c28e0932f6235cdadea946603d73e6fcfbee503e8c905 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:19:14.542 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:14 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[65252]: ts=2024-03-26T04:19:14.140Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-26T04:19:14.542 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:14 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[65252]: ts=2024-03-26T04:19:14.140Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-26T04:19:14.542 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:14 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[65252]: ts=2024-03-26T04:19:14.140Z caller=main.go:570 level=info host_details="(Linux 5.14.0-432.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Mar 20 20:26:59 UTC 2024 x86_64 smithi183 (none))" 2024-03-26T04:19:14.542 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:14 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[65252]: ts=2024-03-26T04:19:14.140Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-26T04:19:14.542 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:14 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[65252]: ts=2024-03-26T04:19:14.140Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-26T04:19:14.542 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:14 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[65252]: ts=2024-03-26T04:19:14.141Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-26T04:19:14.542 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:14 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[65252]: ts=2024-03-26T04:19:14.141Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-26T04:19:14.543 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:14 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[65252]: ts=2024-03-26T04:19:14.142Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-26T04:19:14.543 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:14 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[65252]: ts=2024-03-26T04:19:14.142Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-26T04:19:14.543 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:14 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[65252]: ts=2024-03-26T04:19:14.145Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-26T04:19:14.543 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:14 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[65252]: ts=2024-03-26T04:19:14.145Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.832µs 2024-03-26T04:19:14.543 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:14 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[65252]: ts=2024-03-26T04:19:14.145Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-26T04:19:14.543 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:14 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[65252]: ts=2024-03-26T04:19:14.145Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-03-26T04:19:14.543 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:14 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[65252]: ts=2024-03-26T04:19:14.145Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-03-26T04:19:14.543 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:14 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[65252]: ts=2024-03-26T04:19:14.146Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=23.029µs wal_replay_duration=626.194µs wbl_replay_duration=123ns total_replay_duration=667.375µs 2024-03-26T04:19:14.543 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:14 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[65252]: ts=2024-03-26T04:19:14.146Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-26T04:19:14.543 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:14 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[65252]: ts=2024-03-26T04:19:14.146Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-26T04:19:14.543 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:14 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[65252]: ts=2024-03-26T04:19:14.146Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-26T04:19:14.543 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:14 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[65252]: ts=2024-03-26T04:19:14.164Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=17.965576ms db_storage=698ns remote_storage=1.087µs web_handler=464ns query_engine=430ns scrape=304.982µs scrape_sd=65.342µs notify=9.803µs notify_sd=15.689µs rules=17.341482ms tracing=6.315µs 2024-03-26T04:19:14.544 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:14 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[65252]: ts=2024-03-26T04:19:14.164Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-26T04:19:14.544 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:14 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[65252]: ts=2024-03-26T04:19:14.164Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-26T04:19:14.544 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:14 smithi183 bash[65237]: 12fe18d01c61267dee1c28e0932f6235cdadea946603d73e6fcfbee503e8c905 2024-03-26T04:19:14.544 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:19:14 smithi183 systemd[1]: Started Ceph prometheus.a for d7d59826-eb26-11ee-95d0-87774f69a715. 2024-03-26T04:19:14.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:19:14 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: [26/Mar/2024:04:19:14] ENGINE Bus STOPPING 2024-03-26T04:19:14.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:19:14 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: [26/Mar/2024:04:19:14] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-26T04:19:14.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:19:14 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: [26/Mar/2024:04:19:14] ENGINE Bus STOPPED 2024-03-26T04:19:14.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:19:14 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: [26/Mar/2024:04:19:14] ENGINE Bus STARTING 2024-03-26T04:19:14.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:19:14 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: [26/Mar/2024:04:19:14] ENGINE Serving on http://:::9283 2024-03-26T04:19:14.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:19:14 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: [26/Mar/2024:04:19:14] ENGINE Bus STARTED 2024-03-26T04:19:15.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:15 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T04:19:15.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:15 smithi183 ceph-mon[33745]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T04:19:15.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:15 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi016.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T04:19:15.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:15 smithi183 ceph-mon[33745]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi016.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T04:19:15.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:15 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:15.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:15 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T04:19:15.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:15 smithi183 ceph-mon[33745]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T04:19:15.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:15 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi183.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T04:19:15.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:15 smithi183 ceph-mon[33745]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi183.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T04:19:15.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:15 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:15.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:15 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T04:19:15.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:15 smithi183 ceph-mon[33745]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T04:19:15.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:15 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi183.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T04:19:15.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:15 smithi183 ceph-mon[33745]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi183.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T04:19:15.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:15 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:15.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:15 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:19:15.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:15 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T04:19:15.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:15 smithi016 ceph-mon[37215]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T04:19:15.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:15 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi016.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T04:19:15.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:15 smithi016 ceph-mon[37215]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi016.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T04:19:15.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:15 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:15.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:15 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T04:19:15.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:15 smithi016 ceph-mon[37215]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T04:19:15.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:15 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi183.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T04:19:15.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:15 smithi016 ceph-mon[37215]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi183.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T04:19:15.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:15 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:15.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:15 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T04:19:15.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:15 smithi016 ceph-mon[37215]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T04:19:15.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:15 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi183.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T04:19:15.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:15 smithi016 ceph-mon[37215]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi183.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T04:19:15.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:15 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:15.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:15 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:19:15.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:15 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T04:19:15.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:15 smithi016 ceph-mon[29101]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-26T04:19:15.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:15 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi016.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T04:19:15.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:15 smithi016 ceph-mon[29101]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi016.front.sepia.ceph.com:9093"}]: dispatch 2024-03-26T04:19:15.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:15 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:15.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:15 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T04:19:15.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:15 smithi016 ceph-mon[29101]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-26T04:19:15.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:15 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi183.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T04:19:15.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:15 smithi016 ceph-mon[29101]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi183.front.sepia.ceph.com:9095"}]: dispatch 2024-03-26T04:19:15.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:15 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:15.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:15 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T04:19:15.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:15 smithi016 ceph-mon[29101]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-26T04:19:15.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:15 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi183.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T04:19:15.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:15 smithi016 ceph-mon[29101]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi183.front.sepia.ceph.com:3000"}]: dispatch 2024-03-26T04:19:15.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:15 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:15.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:15 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:19:16.523 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:16 smithi016 ceph-mon[37215]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:19:16.523 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:16 smithi016 ceph-mon[29101]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:19:16.643 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:16 smithi183 ceph-mon[33745]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:19:18.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:18 smithi016 ceph-mon[37215]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:19:18.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:18 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:18.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:18 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:18.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:18 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:18.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:18 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:18.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:18 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:19:18.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:18 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:19:18.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:18 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:18.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:18 smithi016 ceph-mon[29101]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:19:18.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:18 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:18.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:18 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:18.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:18 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:18.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:18 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:18.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:18 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:19:18.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:18 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:19:18.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:18 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:19.044 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:18 smithi183 ceph-mon[33745]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:19:19.044 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:18 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:19.045 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:18 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:19.045 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:18 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:19.045 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:18 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:19.045 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:18 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:19:19.045 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:18 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:19:19.045 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:18 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:19:19.412 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:19:19 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-alertmanager-a[73529]: ts=2024-03-26T04:19:19.121Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003266121s 2024-03-26T04:19:20.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:20 smithi016 ceph-mon[29101]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:19:20.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:20 smithi016 ceph-mon[37215]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:19:21.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:20 smithi183 ceph-mon[33745]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:19:21.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:21 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:19:21.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:21 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:19:22.011 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:21 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:19:22.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:19:22 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:19:22.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:22 smithi016 ceph-mon[29101]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:19:22.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:22 smithi016 ceph-mon[37215]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:19:23.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:22 smithi183 ceph-mon[33745]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:19:23.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:23 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:19:23.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:23 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:19:24.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:23 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:19:24.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:24 smithi016 ceph-mon[29101]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:19:24.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:24 smithi016 ceph-mon[37215]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:19:25.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:24 smithi183 ceph-mon[33745]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:19:26.843 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:26 smithi183 ceph-mon[33745]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:19:26.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:26 smithi016 ceph-mon[37215]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:19:26.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:26 smithi016 ceph-mon[29101]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:19:28.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:19:28 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:19:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:19:28.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:28 smithi016 ceph-mon[37215]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:19:28.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:28 smithi016 ceph-mon[29101]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:19:29.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:28 smithi183 ceph-mon[33745]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:19:30.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:30 smithi016 ceph-mon[29101]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:19:30.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:30 smithi016 ceph-mon[37215]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:19:31.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:30 smithi183 ceph-mon[33745]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:19:32.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:19:32 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:19:32.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:32 smithi016 ceph-mon[29101]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:19:32.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:32 smithi016 ceph-mon[37215]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:19:33.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:32 smithi183 ceph-mon[33745]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:19:33.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:33 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:19:33.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:33 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:19:34.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:33 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:19:34.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:34 smithi016 ceph-mon[37215]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:19:34.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:34 smithi016 ceph-mon[29101]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:19:35.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:34 smithi183 ceph-mon[33745]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:19:36.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:36 smithi016 ceph-mon[37215]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:19:36.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:36 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:19:36.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:36 smithi016 ceph-mon[29101]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:19:36.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:36 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:19:37.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:36 smithi183 ceph-mon[33745]: pgmap v57: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:19:37.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:36 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:19:38.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:38 smithi016 ceph-mon[37215]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:19:38.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:38 smithi016 ceph-mon[29101]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:19:38.913 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:19:38 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:19:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:19:39.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:38 smithi183 ceph-mon[33745]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:19:40.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:40 smithi016 ceph-mon[37215]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:19:40.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:40 smithi016 ceph-mon[29101]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:19:41.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:40 smithi183 ceph-mon[33745]: pgmap v59: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:19:42.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:19:42 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:19:42.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:42 smithi016 ceph-mon[37215]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:19:42.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:42 smithi016 ceph-mon[29101]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:19:43.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:42 smithi183 ceph-mon[33745]: pgmap v60: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:19:43.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:43 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:19:43.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:43 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:19:44.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:43 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:19:44.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:44 smithi016 ceph-mon[29101]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:19:44.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:44 smithi016 ceph-mon[37215]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:19:45.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:44 smithi183 ceph-mon[33745]: pgmap v61: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:19:45.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:45 smithi016 ceph-mon[37215]: pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:19:45.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:45 smithi016 ceph-mon[29101]: pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:19:46.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:45 smithi183 ceph-mon[33745]: pgmap v62: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:19:47.130 INFO:tasks.workunit.client.0.smithi016.stderr:Updating files: 96% (12020/12431) Updating files: 97% (12059/12431) Updating files: 98% (12183/12431) Updating files: 99% (12307/12431) Updating files: 100% (12431/12431) Updating files: 100% (12431/12431), done. 2024-03-26T04:19:47.397 INFO:tasks.workunit.client.0.smithi016.stderr:Note: switching to '9f04c91b77e4619963d41730d8cf756e3a6152de'. 2024-03-26T04:19:47.397 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-03-26T04:19:47.397 INFO:tasks.workunit.client.0.smithi016.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-26T04:19:47.397 INFO:tasks.workunit.client.0.smithi016.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-26T04:19:47.397 INFO:tasks.workunit.client.0.smithi016.stderr:state without impacting any branches by switching back to a branch. 2024-03-26T04:19:47.397 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-03-26T04:19:47.397 INFO:tasks.workunit.client.0.smithi016.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-26T04:19:47.397 INFO:tasks.workunit.client.0.smithi016.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-26T04:19:47.397 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-03-26T04:19:47.397 INFO:tasks.workunit.client.0.smithi016.stderr: git switch -c 2024-03-26T04:19:47.397 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-03-26T04:19:47.398 INFO:tasks.workunit.client.0.smithi016.stderr:Or undo this operation with: 2024-03-26T04:19:47.398 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-03-26T04:19:47.398 INFO:tasks.workunit.client.0.smithi016.stderr: git switch - 2024-03-26T04:19:47.398 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-03-26T04:19:47.398 INFO:tasks.workunit.client.0.smithi016.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-26T04:19:47.398 INFO:tasks.workunit.client.0.smithi016.stderr: 2024-03-26T04:19:47.398 INFO:tasks.workunit.client.0.smithi016.stderr:HEAD is now at 9f04c91b77e Merge PR #56391 into squid 2024-03-26T04:19:47.401 DEBUG:teuthology.orchestra.run.smithi016:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-26T04:19:47.422 INFO:tasks.workunit.client.0.smithi016.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-26T04:19:47.424 INFO:tasks.workunit.client.0.smithi016.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-26T04:19:47.424 INFO:tasks.workunit.client.0.smithi016.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-26T04:19:48.218 INFO:tasks.workunit.client.0.smithi016.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-26T04:19:48.284 INFO:tasks.workunit.client.0.smithi016.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-26T04:19:48.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:48 smithi183 ceph-mon[33745]: pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:19:48.325 INFO:tasks.workunit.client.0.smithi016.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-26T04:19:48.327 INFO:tasks.workunit.client.0.smithi016.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-26T04:19:48.327 INFO:tasks.workunit.client.0.smithi016.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-26T04:19:48.366 INFO:tasks.workunit.client.0.smithi016.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-26T04:19:48.369 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-03-26T04:19:48.369 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-26T04:19:48.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:48 smithi016 ceph-mon[29101]: pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:19:48.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:48 smithi016 ceph-mon[37215]: pgmap v63: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:19:48.425 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-03-26T04:19:48.426 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-03-26T04:19:48.426 DEBUG:teuthology.orchestra.run.smithi016:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9f04c91b77e4619963d41730d8cf756e3a6152de TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-03-26T04:19:48.493 INFO:tasks.workunit.client.0.smithi016.stderr:+ parallel=1 2024-03-26T04:19:48.494 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' '' = --serial ']' 2024-03-26T04:19:48.494 INFO:tasks.workunit.client.0.smithi016.stderr:+ crimson=0 2024-03-26T04:19:48.494 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' '' = --crimson ']' 2024-03-26T04:19:48.494 INFO:tasks.workunit.client.0.smithi016.stderr:+ color= 2024-03-26T04:19:48.494 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -t 1 ']' 2024-03-26T04:19:48.494 INFO:tasks.workunit.client.0.smithi016.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-03-26T04:19:48.494 INFO:tasks.workunit.client.0.smithi016.stderr:+ GTEST_OUTPUT_DIR=/home/ubuntu/cephtest/archive/unit_test_xml_report 2024-03-26T04:19:48.494 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir -p /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-03-26T04:19:48.495 INFO:tasks.workunit.client.0.smithi016.stderr:+ declare -A pids 2024-03-26T04:19:48.495 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:19:48.495 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:19:48.495 INFO:tasks.workunit.client.0.smithi016.stderr:++ printf %25s api_aio 2024-03-26T04:19:48.495 INFO:tasks.workunit.client.0.smithi016.stderr:+ r=' api_aio' 2024-03-26T04:19:48.496 INFO:tasks.workunit.client.0.smithi016.stderr:++ echo api_aio 2024-03-26T04:19:48.496 INFO:tasks.workunit.client.0.smithi016.stderr:++ awk '{print $1}' 2024-03-26T04:19:48.496 INFO:tasks.workunit.client.0.smithi016.stderr:+ ff=api_aio 2024-03-26T04:19:48.496 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74193 2024-03-26T04:19:48.496 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'test api_aio on pid 74193' 2024-03-26T04:19:48.497 INFO:tasks.workunit.client.0.smithi016.stdout:test api_aio on pid 74193 2024-03-26T04:19:48.500 INFO:tasks.workunit.client.0.smithi016.stderr:+ pids[$f]=74193 2024-03-26T04:19:48.500 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:19:48.500 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:19:48.500 INFO:tasks.workunit.client.0.smithi016.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-03-26T04:19:48.500 INFO:tasks.workunit.client.0.smithi016.stderr:++ printf %25s api_aio_pp 2024-03-26T04:19:48.500 INFO:tasks.workunit.client.0.smithi016.stderr:+ r=' api_aio_pp' 2024-03-26T04:19:48.500 INFO:tasks.workunit.client.0.smithi016.stderr:++ echo api_aio_pp 2024-03-26T04:19:48.500 INFO:tasks.workunit.client.0.smithi016.stderr:++ awk '{print $1}' 2024-03-26T04:19:48.500 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:19:48.500 INFO:tasks.workunit.client.0.smithi016.stderr:+ . /etc/bashrc 2024-03-26T04:19:48.500 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -z '' ']' 2024-03-26T04:19:48.500 INFO:tasks.workunit.client.0.smithi016.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:19:48.500 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.500 INFO:tasks.workunit.client.0.smithi016.stderr:++ shopt -q login_shell 2024-03-26T04:19:48.501 INFO:tasks.workunit.client.0.smithi016.stderr:+++ umask 2024-03-26T04:19:48.501 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:19:48.501 INFO:tasks.workunit.client.0.smithi016.stderr:++ SHELL=/bin/bash 2024-03-26T04:19:48.501 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.501 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:19:48.501 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.501 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:19:48.501 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:19:48.501 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.501 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:19:48.501 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.501 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:19:48.501 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.501 INFO:tasks.workunit.client.0.smithi016.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.502 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.502 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:19:48.502 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.502 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:19:48.502 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.502 INFO:tasks.workunit.client.0.smithi016.stderr:+ ff=api_aio_pp 2024-03-26T04:19:48.502 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74200 2024-03-26T04:19:48.502 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'test api_aio_pp on pid 74200' 2024-03-26T04:19:48.502 INFO:tasks.workunit.client.0.smithi016.stdout:test api_aio_pp on pid 74200 2024-03-26T04:19:48.506 INFO:tasks.workunit.client.0.smithi016.stderr:+ pids[$f]=74200 2024-03-26T04:19:48.506 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:19:48.506 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:19:48.506 INFO:tasks.workunit.client.0.smithi016.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-03-26T04:19:48.506 INFO:tasks.workunit.client.0.smithi016.stderr:++ printf %25s api_io 2024-03-26T04:19:48.506 INFO:tasks.workunit.client.0.smithi016.stderr:+ r=' api_io' 2024-03-26T04:19:48.506 INFO:tasks.workunit.client.0.smithi016.stderr:++ echo api_io 2024-03-26T04:19:48.506 INFO:tasks.workunit.client.0.smithi016.stderr:++ awk '{print $1}' 2024-03-26T04:19:48.506 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:19:48.506 INFO:tasks.workunit.client.0.smithi016.stderr:+ . /etc/bashrc 2024-03-26T04:19:48.507 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -z '' ']' 2024-03-26T04:19:48.507 INFO:tasks.workunit.client.0.smithi016.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:19:48.507 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.507 INFO:tasks.workunit.client.0.smithi016.stderr:++ shopt -q login_shell 2024-03-26T04:19:48.507 INFO:tasks.workunit.client.0.smithi016.stderr:+++ umask 2024-03-26T04:19:48.507 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:19:48.507 INFO:tasks.workunit.client.0.smithi016.stderr:++ SHELL=/bin/bash 2024-03-26T04:19:48.507 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.507 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:19:48.507 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.507 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:19:48.507 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:19:48.507 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:19:48.507 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:19:48.508 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:19:48.508 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.508 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:19:48.508 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.508 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:19:48.508 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.508 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.508 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:19:48.508 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:19:48.508 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.508 INFO:tasks.workunit.client.0.smithi016.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.508 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:19:48.508 INFO:tasks.workunit.client.0.smithi016.stderr:+++ return 2024-03-26T04:19:48.508 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.508 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:19:48.509 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.509 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.509 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:19:48.509 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:19:48.509 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.509 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:19:48.509 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.509 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export S_COLORS=auto 2024-03-26T04:19:48.509 INFO:tasks.workunit.client.0.smithi016.stderr:+++ S_COLORS=auto 2024-03-26T04:19:48.509 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.509 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:19:48.509 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.509 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:19:48.509 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.510 INFO:tasks.workunit.client.0.smithi016.stderr:+ ff=api_io 2024-03-26T04:19:48.510 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74209 2024-03-26T04:19:48.510 INFO:tasks.workunit.client.0.smithi016.stdout:test api_io on pid 74209 2024-03-26T04:19:48.514 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'test api_io on pid 74209' 2024-03-26T04:19:48.514 INFO:tasks.workunit.client.0.smithi016.stderr:+ pids[$f]=74209 2024-03-26T04:19:48.514 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:19:48.514 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:19:48.514 INFO:tasks.workunit.client.0.smithi016.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-03-26T04:19:48.514 INFO:tasks.workunit.client.0.smithi016.stderr:++ printf %25s api_io_pp 2024-03-26T04:19:48.514 INFO:tasks.workunit.client.0.smithi016.stderr:+ r=' api_io_pp' 2024-03-26T04:19:48.514 INFO:tasks.workunit.client.0.smithi016.stderr:++ echo api_io_pp 2024-03-26T04:19:48.514 INFO:tasks.workunit.client.0.smithi016.stderr:++ awk '{print $1}' 2024-03-26T04:19:48.514 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:19:48.514 INFO:tasks.workunit.client.0.smithi016.stderr:+ . /etc/bashrc 2024-03-26T04:19:48.515 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -z '' ']' 2024-03-26T04:19:48.515 INFO:tasks.workunit.client.0.smithi016.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:19:48.515 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.515 INFO:tasks.workunit.client.0.smithi016.stderr:++ shopt -q login_shell 2024-03-26T04:19:48.515 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:19:48.515 INFO:tasks.workunit.client.0.smithi016.stderr:+++ umask 2024-03-26T04:19:48.515 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:19:48.515 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:19:48.515 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:19:48.515 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:19:48.515 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:19:48.515 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.515 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:19:48.515 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.515 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:19:48.516 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.516 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:19:48.516 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:19:48.516 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.516 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.516 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:19:48.516 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:19:48.516 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:19:48.516 INFO:tasks.workunit.client.0.smithi016.stderr:++ SHELL=/bin/bash 2024-03-26T04:19:48.516 INFO:tasks.workunit.client.0.smithi016.stderr:+++ return 2024-03-26T04:19:48.516 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.516 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:19:48.516 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.516 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:19:48.516 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export S_COLORS=auto 2024-03-26T04:19:48.517 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.517 INFO:tasks.workunit.client.0.smithi016.stderr:+++ S_COLORS=auto 2024-03-26T04:19:48.517 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.517 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:19:48.517 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:19:48.517 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.517 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.517 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:19:48.517 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:19:48.517 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.517 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:19:48.517 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.517 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:19:48.517 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.518 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:19:48.518 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.518 INFO:tasks.workunit.client.0.smithi016.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.518 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.518 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:19:48.518 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.518 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:19:48.518 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.518 INFO:tasks.workunit.client.0.smithi016.stderr:+ ff=api_io_pp 2024-03-26T04:19:48.518 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74220 2024-03-26T04:19:48.518 INFO:tasks.workunit.client.0.smithi016.stdout:test api_io_pp on pid 74220 2024-03-26T04:19:48.524 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'test api_io_pp on pid 74220' 2024-03-26T04:19:48.524 INFO:tasks.workunit.client.0.smithi016.stderr:+ pids[$f]=74220 2024-03-26T04:19:48.524 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:19:48.524 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:19:48.524 INFO:tasks.workunit.client.0.smithi016.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-03-26T04:19:48.524 INFO:tasks.workunit.client.0.smithi016.stderr:++ printf %25s api_asio 2024-03-26T04:19:48.524 INFO:tasks.workunit.client.0.smithi016.stderr:+ r=' api_asio' 2024-03-26T04:19:48.524 INFO:tasks.workunit.client.0.smithi016.stderr:++ echo api_asio 2024-03-26T04:19:48.524 INFO:tasks.workunit.client.0.smithi016.stderr:++ awk '{print $1}' 2024-03-26T04:19:48.524 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:19:48.524 INFO:tasks.workunit.client.0.smithi016.stderr:+ . /etc/bashrc 2024-03-26T04:19:48.524 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:19:48.525 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:19:48.525 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -z '' ']' 2024-03-26T04:19:48.525 INFO:tasks.workunit.client.0.smithi016.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:19:48.525 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:19:48.525 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.525 INFO:tasks.workunit.client.0.smithi016.stderr:++ shopt -q login_shell 2024-03-26T04:19:48.525 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.525 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:19:48.525 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.525 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:19:48.525 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:19:48.525 INFO:tasks.workunit.client.0.smithi016.stderr:+++ return 2024-03-26T04:19:48.525 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.525 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:19:48.525 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.526 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:19:48.526 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:19:48.526 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export S_COLORS=auto 2024-03-26T04:19:48.526 INFO:tasks.workunit.client.0.smithi016.stderr:+++ S_COLORS=auto 2024-03-26T04:19:48.526 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:19:48.526 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.526 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:19:48.526 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:19:48.526 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.526 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:19:48.526 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:19:48.526 INFO:tasks.workunit.client.0.smithi016.stderr:+++ umask 2024-03-26T04:19:48.526 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:19:48.526 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.527 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.527 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:19:48.527 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:19:48.527 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.527 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:19:48.527 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.527 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:19:48.527 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.527 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:19:48.527 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:19:48.527 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:19:48.527 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:19:48.527 INFO:tasks.workunit.client.0.smithi016.stderr:++ SHELL=/bin/bash 2024-03-26T04:19:48.527 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.528 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:19:48.528 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.528 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.528 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:19:48.528 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.528 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:19:48.528 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.528 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.528 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:19:48.528 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.528 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:19:48.528 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:19:48.528 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup 2024-03-26T04:19:48.528 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:19:48.528 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.529 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:19:48.529 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.529 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:19:48.529 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.529 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/locale 2024-03-26T04:19:48.529 INFO:tasks.workunit.client.0.smithi016.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.529 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.529 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:19:48.529 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.529 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:19:48.529 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.529 INFO:tasks.workunit.client.0.smithi016.stderr:+ ff=api_asio 2024-03-26T04:19:48.529 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74234 2024-03-26T04:19:48.530 INFO:tasks.workunit.client.0.smithi016.stdout:test api_asio on pid 74234 2024-03-26T04:19:48.537 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'test api_asio on pid 74234' 2024-03-26T04:19:48.537 INFO:tasks.workunit.client.0.smithi016.stderr:+ pids[$f]=74234 2024-03-26T04:19:48.537 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:19:48.537 INFO:tasks.workunit.client.0.smithi016.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-03-26T04:19:48.537 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:19:48.537 INFO:tasks.workunit.client.0.smithi016.stderr:++ printf %25s api_list 2024-03-26T04:19:48.537 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.537 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:19:48.537 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.537 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:19:48.538 INFO:tasks.workunit.client.0.smithi016.stderr:+ r=' api_list' 2024-03-26T04:19:48.538 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:19:48.538 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:19:48.538 INFO:tasks.workunit.client.0.smithi016.stderr:++ echo api_list 2024-03-26T04:19:48.538 INFO:tasks.workunit.client.0.smithi016.stderr:++ awk '{print $1}' 2024-03-26T04:19:48.538 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:19:48.538 INFO:tasks.workunit.client.0.smithi016.stderr:+ . /etc/bashrc 2024-03-26T04:19:48.538 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -z '' ']' 2024-03-26T04:19:48.538 INFO:tasks.workunit.client.0.smithi016.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:19:48.538 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.538 INFO:tasks.workunit.client.0.smithi016.stderr:++ shopt -q login_shell 2024-03-26T04:19:48.538 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:19:48.538 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:19:48.538 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:19:48.538 INFO:tasks.workunit.client.0.smithi016.stderr:+++ umask 2024-03-26T04:19:48.539 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.539 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:19:48.539 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.539 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:19:48.539 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:19:48.539 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.539 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:19:48.539 INFO:tasks.workunit.client.0.smithi016.stderr:++ SHELL=/bin/bash 2024-03-26T04:19:48.539 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:19:48.539 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:19:48.539 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:19:48.539 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.539 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.539 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:19:48.540 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:19:48.540 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.540 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.540 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:19:48.540 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:19:48.540 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:19:48.540 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.540 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:19:48.540 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.540 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:19:48.540 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:19:48.540 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.540 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:19:48.540 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.540 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:19:48.541 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.541 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:19:48.541 INFO:tasks.workunit.client.0.smithi016.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.541 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.541 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:19:48.541 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:19:48.541 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.541 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:19:48.541 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.541 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:19:48.541 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:19:48.541 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.541 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.541 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:19:48.541 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.542 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup 2024-03-26T04:19:48.542 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:19:48.542 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.542 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:19:48.542 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:19:48.542 INFO:tasks.workunit.client.0.smithi016.stderr:+++ return 2024-03-26T04:19:48.542 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.542 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:19:48.542 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.542 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:19:48.542 INFO:tasks.workunit.client.0.smithi016.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:19:48.542 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export S_COLORS=auto 2024-03-26T04:19:48.542 INFO:tasks.workunit.client.0.smithi016.stderr:+++ S_COLORS=auto 2024-03-26T04:19:48.542 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.543 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:19:48.543 INFO:tasks.workunit.client.0.smithi016.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:19:48.543 INFO:tasks.workunit.client.0.smithi016.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:19:48.543 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.543 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:19:48.543 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.543 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.543 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:19:48.543 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.543 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:19:48.543 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/locale 2024-03-26T04:19:48.543 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup config 2024-03-26T04:19:48.543 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.543 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.543 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:19:48.544 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.544 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:19:48.544 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.544 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:19:48.544 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:19:48.544 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.544 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:19:48.544 INFO:tasks.workunit.client.0.smithi016.stderr:+ ff=api_list 2024-03-26T04:19:48.544 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.544 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:19:48.544 INFO:tasks.workunit.client.0.smithi016.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:19:48.544 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74250 2024-03-26T04:19:48.545 INFO:tasks.workunit.client.0.smithi016.stdout:test api_list on pid 74250 2024-03-26T04:19:48.550 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'test api_list on pid 74250' 2024-03-26T04:19:48.550 INFO:tasks.workunit.client.0.smithi016.stderr:+ pids[$f]=74250 2024-03-26T04:19:48.550 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:19:48.550 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:19:48.550 INFO:tasks.workunit.client.0.smithi016.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-03-26T04:19:48.550 INFO:tasks.workunit.client.0.smithi016.stderr:++ printf %25s api_lock 2024-03-26T04:19:48.551 INFO:tasks.workunit.client.0.smithi016.stderr:+++++ readlink /proc/74193/exe 2024-03-26T04:19:48.551 INFO:tasks.workunit.client.0.smithi016.stderr:+ r=' api_lock' 2024-03-26T04:19:48.551 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.551 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:19:48.551 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.551 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:19:48.551 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:19:48.551 INFO:tasks.workunit.client.0.smithi016.stderr:++ echo api_lock 2024-03-26T04:19:48.551 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:19:48.551 INFO:tasks.workunit.client.0.smithi016.stderr:++ awk '{print $1}' 2024-03-26T04:19:48.551 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:19:48.551 INFO:tasks.workunit.client.0.smithi016.stderr:++++ basename /usr/bin/bash 2024-03-26T04:19:48.551 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:19:48.551 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:19:48.552 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.552 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:19:48.552 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.552 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:19:48.552 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:19:48.552 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:19:48.552 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.552 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:19:48.552 INFO:tasks.workunit.client.0.smithi016.stderr:+ . /etc/bashrc 2024-03-26T04:19:48.552 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.552 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:19:48.552 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.552 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:19:48.552 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -z '' ']' 2024-03-26T04:19:48.552 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.553 INFO:tasks.workunit.client.0.smithi016.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:19:48.553 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:19:48.553 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.553 INFO:tasks.workunit.client.0.smithi016.stderr:++ shopt -q login_shell 2024-03-26T04:19:48.553 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup 2024-03-26T04:19:48.553 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:19:48.553 INFO:tasks.workunit.client.0.smithi016.stderr:+++ umask 2024-03-26T04:19:48.553 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:19:48.553 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:19:48.553 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.553 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:19:48.553 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.553 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:19:48.553 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/locale 2024-03-26T04:19:48.554 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_declare='declare -f' 2024-03-26T04:19:48.554 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_opt=-f 2024-03-26T04:19:48.554 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:19:48.554 INFO:tasks.workunit.client.0.smithi016.stderr:+++ return 2024-03-26T04:19:48.554 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.554 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export which_declare 2024-03-26T04:19:48.554 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:19:48.554 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:19:48.554 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export -f which 2024-03-26T04:19:48.554 INFO:tasks.workunit.client.0.smithi016.stderr:++ SHELL=/bin/bash 2024-03-26T04:19:48.554 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.554 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:19:48.554 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset i 2024-03-26T04:19:48.554 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset -f pathmunge 2024-03-26T04:19:48.555 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export S_COLORS=auto 2024-03-26T04:19:48.555 INFO:tasks.workunit.client.0.smithi016.stderr:+++ S_COLORS=auto 2024-03-26T04:19:48.555 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.555 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.555 INFO:tasks.workunit.client.0.smithi016.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:19:48.555 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:19:48.555 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:19:48.555 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.555 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.555 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:19:48.555 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:19:48.555 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.555 INFO:tasks.workunit.client.0.smithi016.stderr:+ export PATH 2024-03-26T04:19:48.555 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:19:48.555 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:19:48.556 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.556 INFO:tasks.workunit.client.0.smithi016.stderr:+ unset rc 2024-03-26T04:19:48.556 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:19:48.556 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.556 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:19:48.556 INFO:tasks.workunit.client.0.smithi016.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:19:48.556 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.556 INFO:tasks.workunit.client.0.smithi016.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.556 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.556 INFO:tasks.workunit.client.0.smithi016.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:19:48.556 INFO:tasks.workunit.client.0.smithi016.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:19:48.556 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:19:48.556 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.556 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:19:48.556 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.557 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:19:48.557 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.557 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.557 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:19:48.557 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup config 2024-03-26T04:19:48.557 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.557 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2024-03-26T04:19:48.557 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.557 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:19:48.557 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.557 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:19:48.557 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.557 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:19:48.557 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:19:48.557 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:19:48.558 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.558 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:19:48.558 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.558 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:19:48.558 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:19:48.558 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:19:48.558 INFO:tasks.workunit.client.0.smithi016.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:19:48.558 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.558 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:19:48.558 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.558 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:19:48.558 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:19:48.558 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.558 INFO:tasks.workunit.client.0.smithi016.stderr:+ ff=api_lock 2024-03-26T04:19:48.558 INFO:tasks.workunit.client.0.smithi016.stderr:+ sed 's/^/ api_aio: /' 2024-03-26T04:19:48.559 INFO:tasks.workunit.client.0.smithi016.stdout:test api_lock on pid 74270 2024-03-26T04:19:48.564 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74270 2024-03-26T04:19:48.564 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'test api_lock on pid 74270' 2024-03-26T04:19:48.564 INFO:tasks.workunit.client.0.smithi016.stderr:+ pids[$f]=74270 2024-03-26T04:19:48.564 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.564 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:19:48.564 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:19:48.564 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:19:48.564 INFO:tasks.workunit.client.0.smithi016.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-03-26T04:19:48.564 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.565 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:19:48.565 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:19:48.565 INFO:tasks.workunit.client.0.smithi016.stderr:+++++ readlink /proc/74200/exe 2024-03-26T04:19:48.565 INFO:tasks.workunit.client.0.smithi016.stderr:++ printf %25s api_lock_pp 2024-03-26T04:19:48.565 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:19:48.565 INFO:tasks.workunit.client.0.smithi016.stderr:+ r=' api_lock_pp' 2024-03-26T04:19:48.565 INFO:tasks.workunit.client.0.smithi016.stderr:+ tee ceph_test_rados_api_aio.log 2024-03-26T04:19:48.565 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:19:48.565 INFO:tasks.workunit.client.0.smithi016.stderr:+ . /etc/bashrc 2024-03-26T04:19:48.565 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -z '' ']' 2024-03-26T04:19:48.565 INFO:tasks.workunit.client.0.smithi016.stderr:++++ basename /usr/bin/bash 2024-03-26T04:19:48.565 INFO:tasks.workunit.client.0.smithi016.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:19:48.565 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.565 INFO:tasks.workunit.client.0.smithi016.stderr:++ shopt -q login_shell 2024-03-26T04:19:48.566 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:19:48.566 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:19:48.566 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:19:48.566 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.566 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:19:48.566 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:19:48.566 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.566 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:19:48.566 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:19:48.566 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:19:48.566 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:19:48.566 INFO:tasks.workunit.client.0.smithi016.stderr:+++ umask 2024-03-26T04:19:48.566 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.566 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.567 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:19:48.567 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.567 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:19:48.567 INFO:tasks.workunit.client.0.smithi016.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:19:48.567 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:19:48.567 INFO:tasks.workunit.client.0.smithi016.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:19:48.567 INFO:tasks.workunit.client.0.smithi016.stderr:+++ return 2024-03-26T04:19:48.567 INFO:tasks.workunit.client.0.smithi016.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:19:48.567 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.567 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:19:48.567 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.567 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.567 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:19:48.567 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:19:48.567 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export S_COLORS=auto 2024-03-26T04:19:48.568 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.568 INFO:tasks.workunit.client.0.smithi016.stderr:+++ S_COLORS=auto 2024-03-26T04:19:48.568 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:19:48.568 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.568 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:19:48.568 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup config 2024-03-26T04:19:48.568 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.568 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:19:48.568 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:19:48.568 INFO:tasks.workunit.client.0.smithi016.stderr:++ SHELL=/bin/bash 2024-03-26T04:19:48.568 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.568 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.568 INFO:tasks.workunit.client.0.smithi016.stderr:++ echo api_lock_pp 2024-03-26T04:19:48.569 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_declare='declare -f' 2024-03-26T04:19:48.569 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.569 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_opt=-f 2024-03-26T04:19:48.569 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:19:48.569 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.569 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.569 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:19:48.569 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:19:48.569 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export which_declare 2024-03-26T04:19:48.569 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.569 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export -f which 2024-03-26T04:19:48.569 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:19:48.569 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:19:48.569 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.569 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:19:48.570 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset i 2024-03-26T04:19:48.570 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset -f pathmunge 2024-03-26T04:19:48.570 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:19:48.570 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:19:48.570 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:19:48.570 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:19:48.570 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.570 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.570 INFO:tasks.workunit.client.0.smithi016.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:19:48.570 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:19:48.570 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:19:48.570 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.570 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.571 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:19:48.571 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.571 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:19:48.571 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:19:48.571 INFO:tasks.workunit.client.0.smithi016.stderr:++ awk '{print $1}' 2024-03-26T04:19:48.571 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.571 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:19:48.571 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.571 INFO:tasks.workunit.client.0.smithi016.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.571 INFO:tasks.workunit.client.0.smithi016.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:19:48.571 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.571 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:19:48.571 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.571 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:19:48.571 INFO:tasks.workunit.client.0.smithi016.stderr:+ export PATH 2024-03-26T04:19:48.572 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.572 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:19:48.572 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:19:48.572 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.572 INFO:tasks.workunit.client.0.smithi016.stderr:+ unset rc 2024-03-26T04:19:48.572 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:19:48.572 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.572 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:19:48.572 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.572 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:19:48.572 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.572 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:19:48.572 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup 2024-03-26T04:19:48.573 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2024-03-26T04:19:48.573 INFO:tasks.workunit.client.0.smithi016.stderr:+++++ readlink /proc/74209/exe 2024-03-26T04:19:48.573 INFO:tasks.workunit.client.0.smithi016.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-03-26T04:19:48.573 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/locale 2024-03-26T04:19:48.573 INFO:tasks.workunit.client.0.smithi016.stderr:++++ basename /usr/bin/bash 2024-03-26T04:19:48.573 INFO:tasks.workunit.client.0.smithi016.stderr:+ sed 's/^/ api_aio_pp: /' 2024-03-26T04:19:48.573 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.573 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:19:48.573 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.573 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:19:48.573 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:19:48.573 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_declare='declare -f' 2024-03-26T04:19:48.573 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_opt=-f 2024-03-26T04:19:48.573 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:19:48.574 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export which_declare 2024-03-26T04:19:48.574 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export -f which 2024-03-26T04:19:48.574 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset i 2024-03-26T04:19:48.574 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset -f pathmunge 2024-03-26T04:19:48.574 INFO:tasks.workunit.client.0.smithi016.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:19:48.574 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:19:48.574 INFO:tasks.workunit.client.0.smithi016.stderr:+ export PATH 2024-03-26T04:19:48.574 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:19:48.574 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:19:48.574 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:19:48.574 INFO:tasks.workunit.client.0.smithi016.stderr:+ unset rc 2024-03-26T04:19:48.574 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.574 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:19:48.574 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.575 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:19:48.575 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:19:48.575 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.575 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:19:48.575 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.575 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:19:48.575 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.575 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:19:48.575 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.575 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:19:48.575 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup 2024-03-26T04:19:48.575 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:19:48.575 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:19:48.575 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:19:48.575 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2024-03-26T04:19:48.576 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.576 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:19:48.576 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.576 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:19:48.576 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:19:48.576 INFO:tasks.workunit.client.0.smithi016.stderr:+++ return 2024-03-26T04:19:48.576 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.576 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:19:48.576 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.576 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:19:48.576 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export S_COLORS=auto 2024-03-26T04:19:48.576 INFO:tasks.workunit.client.0.smithi016.stderr:+++ S_COLORS=auto 2024-03-26T04:19:48.576 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.576 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:19:48.577 INFO:tasks.workunit.client.0.smithi016.stderr:+ sed 's/^/ api_io: /' 2024-03-26T04:19:48.577 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.577 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:19:48.577 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.577 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/locale 2024-03-26T04:19:48.577 INFO:tasks.workunit.client.0.smithi016.stderr:+ tee ceph_test_rados_api_io.log 2024-03-26T04:19:48.577 INFO:tasks.workunit.client.0.smithi016.stdout:test api_lock_pp on pid 74301 2024-03-26T04:19:48.580 INFO:tasks.workunit.client.0.smithi016.stderr:+ ff=api_lock_pp 2024-03-26T04:19:48.580 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74301 2024-03-26T04:19:48.580 INFO:tasks.workunit.client.0.smithi016.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:19:48.580 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'test api_lock_pp on pid 74301' 2024-03-26T04:19:48.580 INFO:tasks.workunit.client.0.smithi016.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:19:48.580 INFO:tasks.workunit.client.0.smithi016.stderr:+ pids[$f]=74301 2024-03-26T04:19:48.580 INFO:tasks.workunit.client.0.smithi016.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:19:48.580 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:19:48.580 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:19:48.580 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.580 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:19:48.580 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.581 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:19:48.581 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup config 2024-03-26T04:19:48.581 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.581 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:19:48.581 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.581 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:19:48.581 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:19:48.581 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.581 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:19:48.581 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:19:48.581 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.581 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:19:48.581 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.581 INFO:tasks.workunit.client.0.smithi016.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-03-26T04:19:48.582 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:19:48.582 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.582 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:19:48.582 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:19:48.582 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.582 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:19:48.582 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:19:48.582 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.582 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:19:48.582 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.582 INFO:tasks.workunit.client.0.smithi016.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:19:48.582 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.582 INFO:tasks.workunit.client.0.smithi016.stderr:+++++ readlink /proc/74220/exe 2024-03-26T04:19:48.582 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:19:48.583 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.583 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:19:48.583 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:19:48.583 INFO:tasks.workunit.client.0.smithi016.stderr:++ printf %25s api_misc 2024-03-26T04:19:48.583 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:19:48.583 INFO:tasks.workunit.client.0.smithi016.stderr:+ r=' api_misc' 2024-03-26T04:19:48.583 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:19:48.583 INFO:tasks.workunit.client.0.smithi016.stderr:+ . /etc/bashrc 2024-03-26T04:19:48.583 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -z '' ']' 2024-03-26T04:19:48.583 INFO:tasks.workunit.client.0.smithi016.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:19:48.583 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.583 INFO:tasks.workunit.client.0.smithi016.stderr:++++ basename /usr/bin/bash 2024-03-26T04:19:48.583 INFO:tasks.workunit.client.0.smithi016.stderr:++ shopt -q login_shell 2024-03-26T04:19:48.583 INFO:tasks.workunit.client.0.smithi016.stderr:+++ umask 2024-03-26T04:19:48.584 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_declare='declare -f' 2024-03-26T04:19:48.584 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_opt=-f 2024-03-26T04:19:48.584 INFO:tasks.workunit.client.0.smithi016.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:19:48.584 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export which_declare 2024-03-26T04:19:48.584 INFO:tasks.workunit.client.0.smithi016.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:19:48.584 INFO:tasks.workunit.client.0.smithi016.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:19:48.584 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export -f which 2024-03-26T04:19:48.584 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.584 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset i 2024-03-26T04:19:48.584 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset -f pathmunge 2024-03-26T04:19:48.584 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:19:48.584 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:19:48.584 INFO:tasks.workunit.client.0.smithi016.stderr:++ SHELL=/bin/bash 2024-03-26T04:19:48.584 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.584 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:19:48.585 INFO:tasks.workunit.client.0.smithi016.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:19:48.585 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup config 2024-03-26T04:19:48.585 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.585 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.585 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:19:48.585 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.585 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:19:48.585 INFO:tasks.workunit.client.0.smithi016.stderr:+ export PATH 2024-03-26T04:19:48.585 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.585 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:19:48.585 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:19:48.586 INFO:tasks.workunit.client.0.smithi016.stderr:+ unset rc 2024-03-26T04:19:48.586 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.586 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:19:48.586 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.586 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:19:48.586 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:19:48.586 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.586 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:19:48.586 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.586 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:19:48.586 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:19:48.586 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.586 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.587 INFO:tasks.workunit.client.0.smithi016.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.587 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:19:48.587 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.587 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.587 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:19:48.587 INFO:tasks.workunit.client.0.smithi016.stderr:++ echo api_misc 2024-03-26T04:19:48.587 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:19:48.587 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.587 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:19:48.587 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.587 INFO:tasks.workunit.client.0.smithi016.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:19:48.587 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:19:48.587 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:19:48.587 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:19:48.588 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.588 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:19:48.588 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.588 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:19:48.588 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:19:48.588 INFO:tasks.workunit.client.0.smithi016.stderr:++ awk '{print $1}' 2024-03-26T04:19:48.588 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.588 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:19:48.588 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.588 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:19:48.588 INFO:tasks.workunit.client.0.smithi016.stderr:+ sed 's/^/ api_io_pp: /' 2024-03-26T04:19:48.588 INFO:tasks.workunit.client.0.smithi016.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-03-26T04:19:48.588 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.588 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:19:48.588 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.589 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:19:48.589 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup 2024-03-26T04:19:48.589 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/locale 2024-03-26T04:19:48.589 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2024-03-26T04:19:48.589 INFO:tasks.workunit.client.0.smithi016.stderr:+++++ readlink /proc/74234/exe 2024-03-26T04:19:48.589 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.589 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:19:48.589 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.589 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:19:48.589 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:19:48.589 INFO:tasks.workunit.client.0.smithi016.stdout:test api_misc on pid 74323 2024-03-26T04:19:48.592 INFO:tasks.workunit.client.0.smithi016.stderr:+ ff=api_misc 2024-03-26T04:19:48.593 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:19:48.593 INFO:tasks.workunit.client.0.smithi016.stderr:++++ basename /usr/bin/bash 2024-03-26T04:19:48.593 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74323 2024-03-26T04:19:48.593 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'test api_misc on pid 74323' 2024-03-26T04:19:48.593 INFO:tasks.workunit.client.0.smithi016.stderr:+ pids[$f]=74323 2024-03-26T04:19:48.593 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:19:48.593 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:19:48.593 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:19:48.593 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:19:48.593 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:19:48.593 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.593 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:19:48.593 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.594 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:19:48.594 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_declare='declare -f' 2024-03-26T04:19:48.594 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_opt=-f 2024-03-26T04:19:48.594 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:19:48.594 INFO:tasks.workunit.client.0.smithi016.stderr:+++ return 2024-03-26T04:19:48.594 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.594 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:19:48.594 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export which_declare 2024-03-26T04:19:48.594 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.594 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:19:48.594 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export -f which 2024-03-26T04:19:48.594 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset i 2024-03-26T04:19:48.594 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export S_COLORS=auto 2024-03-26T04:19:48.594 INFO:tasks.workunit.client.0.smithi016.stderr:+++ S_COLORS=auto 2024-03-26T04:19:48.595 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset -f pathmunge 2024-03-26T04:19:48.595 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.595 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:19:48.595 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.595 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:19:48.595 INFO:tasks.workunit.client.0.smithi016.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:19:48.595 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.595 INFO:tasks.workunit.client.0.smithi016.stderr:+ export PATH 2024-03-26T04:19:48.595 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:19:48.595 INFO:tasks.workunit.client.0.smithi016.stderr:+ unset rc 2024-03-26T04:19:48.595 INFO:tasks.workunit.client.0.smithi016.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:19:48.595 INFO:tasks.workunit.client.0.smithi016.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:19:48.595 INFO:tasks.workunit.client.0.smithi016.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:19:48.595 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.596 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:19:48.596 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.596 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:19:48.596 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup config 2024-03-26T04:19:48.596 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.596 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.596 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:19:48.596 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.596 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:19:48.596 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.596 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:19:48.596 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:19:48.596 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.596 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:19:48.597 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.597 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:19:48.597 INFO:tasks.workunit.client.0.smithi016.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:19:48.597 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:19:48.597 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:19:48.597 INFO:tasks.workunit.client.0.smithi016.stderr:++ printf %25s api_misc_pp 2024-03-26T04:19:48.597 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:19:48.597 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2024-03-26T04:19:48.597 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.597 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:19:48.597 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.597 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:19:48.597 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:19:48.597 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.598 INFO:tasks.workunit.client.0.smithi016.stderr:+ r=' api_misc_pp' 2024-03-26T04:19:48.598 INFO:tasks.workunit.client.0.smithi016.stderr:+ tee ceph_test_rados_api_asio.log 2024-03-26T04:19:48.598 INFO:tasks.workunit.client.0.smithi016.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-03-26T04:19:48.598 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:19:48.598 INFO:tasks.workunit.client.0.smithi016.stderr:+ . /etc/bashrc 2024-03-26T04:19:48.598 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -z '' ']' 2024-03-26T04:19:48.598 INFO:tasks.workunit.client.0.smithi016.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:19:48.598 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.598 INFO:tasks.workunit.client.0.smithi016.stderr:++ shopt -q login_shell 2024-03-26T04:19:48.598 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:19:48.598 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:19:48.598 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:19:48.598 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.598 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:19:48.599 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.599 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:19:48.599 INFO:tasks.workunit.client.0.smithi016.stderr:+ sed 's/^/ api_asio: /' 2024-03-26T04:19:48.599 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:19:48.599 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.599 INFO:tasks.workunit.client.0.smithi016.stderr:+++ umask 2024-03-26T04:19:48.599 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:19:48.599 INFO:tasks.workunit.client.0.smithi016.stderr:++ SHELL=/bin/bash 2024-03-26T04:19:48.599 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.599 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:19:48.599 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.599 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:19:48.599 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:19:48.599 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.599 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:19:48.600 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.600 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:19:48.600 INFO:tasks.workunit.client.0.smithi016.stderr:++ echo api_misc_pp 2024-03-26T04:19:48.600 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.600 INFO:tasks.workunit.client.0.smithi016.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.600 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.600 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:19:48.600 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.600 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:19:48.600 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.600 INFO:tasks.workunit.client.0.smithi016.stderr:++ awk '{print $1}' 2024-03-26T04:19:48.600 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:19:48.600 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:19:48.600 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:19:48.601 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:19:48.601 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.601 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:19:48.601 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.601 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:19:48.601 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:19:48.601 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:19:48.601 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.601 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:19:48.601 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:19:48.601 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.601 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:19:48.601 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.601 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:19:48.601 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.602 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:19:48.602 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:19:48.602 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.602 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:19:48.602 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.602 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:19:48.602 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.602 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:19:48.602 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.602 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:19:48.602 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.602 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup 2024-03-26T04:19:48.602 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:19:48.602 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.603 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:19:48.603 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup 2024-03-26T04:19:48.603 INFO:tasks.workunit.client.0.smithi016.stderr:+ ff=api_misc_pp 2024-03-26T04:19:48.603 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/locale 2024-03-26T04:19:48.603 INFO:tasks.workunit.client.0.smithi016.stdout:test api_misc_pp on pid 74343 2024-03-26T04:19:48.607 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74343 2024-03-26T04:19:48.607 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'test api_misc_pp on pid 74343' 2024-03-26T04:19:48.607 INFO:tasks.workunit.client.0.smithi016.stderr:+ pids[$f]=74343 2024-03-26T04:19:48.607 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:19:48.607 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/locale 2024-03-26T04:19:48.607 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:19:48.608 INFO:tasks.workunit.client.0.smithi016.stderr:++ printf %25s api_tier_pp 2024-03-26T04:19:48.608 INFO:tasks.workunit.client.0.smithi016.stderr:+ r=' api_tier_pp' 2024-03-26T04:19:48.608 INFO:tasks.workunit.client.0.smithi016.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-03-26T04:19:48.608 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.608 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:19:48.608 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.608 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:19:48.608 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:19:48.608 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:19:48.608 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:19:48.608 INFO:tasks.workunit.client.0.smithi016.stderr:+ . /etc/bashrc 2024-03-26T04:19:48.608 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -z '' ']' 2024-03-26T04:19:48.608 INFO:tasks.workunit.client.0.smithi016.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:19:48.608 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.608 INFO:tasks.workunit.client.0.smithi016.stderr:++ shopt -q login_shell 2024-03-26T04:19:48.609 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.609 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:19:48.609 INFO:tasks.workunit.client.0.smithi016.stderr:+++ umask 2024-03-26T04:19:48.609 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.609 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:19:48.609 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:19:48.609 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:19:48.609 INFO:tasks.workunit.client.0.smithi016.stderr:++ SHELL=/bin/bash 2024-03-26T04:19:48.609 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:19:48.609 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.609 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:19:48.609 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.609 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:19:48.609 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:19:48.610 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.610 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:19:48.610 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.610 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:19:48.610 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.610 INFO:tasks.workunit.client.0.smithi016.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:19:48.610 INFO:tasks.workunit.client.0.smithi016.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:19:48.610 INFO:tasks.workunit.client.0.smithi016.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:19:48.610 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.610 INFO:tasks.workunit.client.0.smithi016.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.610 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:19:48.610 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.610 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:19:48.610 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.611 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.611 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:19:48.611 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:19:48.611 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup config 2024-03-26T04:19:48.611 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.611 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.611 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.611 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:19:48.611 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.611 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:19:48.611 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.611 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:19:48.611 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:19:48.611 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.611 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:19:48.612 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.612 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:19:48.612 INFO:tasks.workunit.client.0.smithi016.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:19:48.612 INFO:tasks.workunit.client.0.smithi016.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:19:48.612 INFO:tasks.workunit.client.0.smithi016.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:19:48.612 INFO:tasks.workunit.client.0.smithi016.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:19:48.612 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.612 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:19:48.612 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.612 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:19:48.612 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup config 2024-03-26T04:19:48.612 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.612 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.612 INFO:tasks.workunit.client.0.smithi016.stderr:++ echo api_tier_pp 2024-03-26T04:19:48.613 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:19:48.613 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.613 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:19:48.613 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.613 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:19:48.613 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:19:48.613 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.613 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:19:48.613 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.613 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:19:48.613 INFO:tasks.workunit.client.0.smithi016.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:19:48.613 INFO:tasks.workunit.client.0.smithi016.stderr:++ awk '{print $1}' 2024-03-26T04:19:48.613 INFO:tasks.workunit.client.0.smithi016.stderr:+++++ readlink /proc/74301/exe 2024-03-26T04:19:48.613 INFO:tasks.workunit.client.0.smithi016.stderr:+++++ readlink /proc/74270/exe 2024-03-26T04:19:48.613 INFO:tasks.workunit.client.0.smithi016.stderr:+++++ readlink /proc/74250/exe 2024-03-26T04:19:48.614 INFO:tasks.workunit.client.0.smithi016.stderr:++++ basename /usr/bin/bash 2024-03-26T04:19:48.614 INFO:tasks.workunit.client.0.smithi016.stderr:++++ basename /usr/bin/bash 2024-03-26T04:19:48.614 INFO:tasks.workunit.client.0.smithi016.stdout:test api_tier_pp on pid 74360 2024-03-26T04:19:48.617 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_declare='declare -f' 2024-03-26T04:19:48.617 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_opt=-f 2024-03-26T04:19:48.617 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export which_declare 2024-03-26T04:19:48.617 INFO:tasks.workunit.client.0.smithi016.stderr:+ ff=api_tier_pp 2024-03-26T04:19:48.617 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export -f which 2024-03-26T04:19:48.617 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset i 2024-03-26T04:19:48.618 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset -f pathmunge 2024-03-26T04:19:48.618 INFO:tasks.workunit.client.0.smithi016.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:19:48.618 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:19:48.618 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74360 2024-03-26T04:19:48.618 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:19:48.618 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'test api_tier_pp on pid 74360' 2024-03-26T04:19:48.618 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:19:48.618 INFO:tasks.workunit.client.0.smithi016.stderr:+ pids[$f]=74360 2024-03-26T04:19:48.618 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:19:48.618 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:19:48.618 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.618 INFO:tasks.workunit.client.0.smithi016.stderr:+ export PATH 2024-03-26T04:19:48.618 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:19:48.618 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.619 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:19:48.619 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:19:48.619 INFO:tasks.workunit.client.0.smithi016.stderr:+ unset rc 2024-03-26T04:19:48.619 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:19:48.619 INFO:tasks.workunit.client.0.smithi016.stderr:+++ return 2024-03-26T04:19:48.619 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.619 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:19:48.619 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.619 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:19:48.619 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export S_COLORS=auto 2024-03-26T04:19:48.619 INFO:tasks.workunit.client.0.smithi016.stderr:+++ S_COLORS=auto 2024-03-26T04:19:48.619 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.619 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:19:48.619 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.619 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:19:48.620 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.620 INFO:tasks.workunit.client.0.smithi016.stderr:++ printf %25s api_pool 2024-03-26T04:19:48.620 INFO:tasks.workunit.client.0.smithi016.stderr:+ r=' api_pool' 2024-03-26T04:19:48.620 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2024-03-26T04:19:48.620 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_declare='declare -f' 2024-03-26T04:19:48.620 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_opt=-f 2024-03-26T04:19:48.620 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export which_declare 2024-03-26T04:19:48.620 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export -f which 2024-03-26T04:19:48.620 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset i 2024-03-26T04:19:48.620 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset -f pathmunge 2024-03-26T04:19:48.620 INFO:tasks.workunit.client.0.smithi016.stderr:+ sed 's/^/ api_lock_pp: /' 2024-03-26T04:19:48.620 INFO:tasks.workunit.client.0.smithi016.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:19:48.620 INFO:tasks.workunit.client.0.smithi016.stderr:+ export PATH 2024-03-26T04:19:48.620 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:19:48.621 INFO:tasks.workunit.client.0.smithi016.stderr:+ unset rc 2024-03-26T04:19:48.621 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2024-03-26T04:19:48.621 INFO:tasks.workunit.client.0.smithi016.stderr:+ sed 's/^/ api_lock: /' 2024-03-26T04:19:48.621 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:19:48.621 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:19:48.621 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:19:48.621 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.621 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:19:48.621 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.621 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:19:48.621 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:19:48.621 INFO:tasks.workunit.client.0.smithi016.stderr:+++ return 2024-03-26T04:19:48.621 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.621 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:19:48.621 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.622 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:19:48.622 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export S_COLORS=auto 2024-03-26T04:19:48.622 INFO:tasks.workunit.client.0.smithi016.stderr:+++ S_COLORS=auto 2024-03-26T04:19:48.622 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.622 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:19:48.622 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.622 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:19:48.622 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.622 INFO:tasks.workunit.client.0.smithi016.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-03-26T04:19:48.622 INFO:tasks.workunit.client.0.smithi016.stderr:+ tee ceph_test_rados_api_lock.log 2024-03-26T04:19:48.622 INFO:tasks.workunit.client.0.smithi016.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-03-26T04:19:48.622 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:19:48.622 INFO:tasks.workunit.client.0.smithi016.stderr:+ . /etc/bashrc 2024-03-26T04:19:48.622 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -z '' ']' 2024-03-26T04:19:48.623 INFO:tasks.workunit.client.0.smithi016.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:19:48.623 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.623 INFO:tasks.workunit.client.0.smithi016.stderr:++ shopt -q login_shell 2024-03-26T04:19:48.623 INFO:tasks.workunit.client.0.smithi016.stderr:++++ basename /usr/bin/bash 2024-03-26T04:19:48.623 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:19:48.623 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:19:48.623 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:19:48.623 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.623 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:19:48.623 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.623 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:19:48.623 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:19:48.623 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.623 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_declare='declare -f' 2024-03-26T04:19:48.623 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_opt=-f 2024-03-26T04:19:48.624 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export which_declare 2024-03-26T04:19:48.624 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export -f which 2024-03-26T04:19:48.624 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset i 2024-03-26T04:19:48.624 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset -f pathmunge 2024-03-26T04:19:48.624 INFO:tasks.workunit.client.0.smithi016.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:19:48.624 INFO:tasks.workunit.client.0.smithi016.stderr:+ export PATH 2024-03-26T04:19:48.624 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:19:48.624 INFO:tasks.workunit.client.0.smithi016.stderr:+ unset rc 2024-03-26T04:19:48.624 INFO:tasks.workunit.client.0.smithi016.stderr:+ tee ceph_test_rados_api_list.log 2024-03-26T04:19:48.624 INFO:tasks.workunit.client.0.smithi016.stderr:++ awk '{print $1}' 2024-03-26T04:19:48.624 INFO:tasks.workunit.client.0.smithi016.stderr:+ sed 's/^/ api_list: /' 2024-03-26T04:19:48.624 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:19:48.624 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:19:48.624 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:19:48.624 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.625 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:19:48.625 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.625 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:19:48.625 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:19:48.625 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.625 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:19:48.625 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.625 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:19:48.625 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.625 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:19:48.625 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.625 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:19:48.625 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup 2024-03-26T04:19:48.625 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/locale 2024-03-26T04:19:48.625 INFO:tasks.workunit.client.0.smithi016.stderr:+++ umask 2024-03-26T04:19:48.626 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:19:48.626 INFO:tasks.workunit.client.0.smithi016.stderr:++ SHELL=/bin/bash 2024-03-26T04:19:48.626 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2024-03-26T04:19:48.626 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.626 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:19:48.626 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.626 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:19:48.626 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:19:48.626 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.626 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:19:48.626 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.626 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:19:48.626 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.626 INFO:tasks.workunit.client.0.smithi016.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.627 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.627 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:19:48.627 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.627 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:19:48.627 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.627 INFO:tasks.workunit.client.0.smithi016.stderr:++ echo api_pool 2024-03-26T04:19:48.627 INFO:tasks.workunit.client.0.smithi016.stdout:test api_pool on pid 74399 2024-03-26T04:19:48.629 INFO:tasks.workunit.client.0.smithi016.stderr:+ ff=api_pool 2024-03-26T04:19:48.629 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74399 2024-03-26T04:19:48.629 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'test api_pool on pid 74399' 2024-03-26T04:19:48.630 INFO:tasks.workunit.client.0.smithi016.stderr:+ pids[$f]=74399 2024-03-26T04:19:48.630 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:19:48.630 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:19:48.630 INFO:tasks.workunit.client.0.smithi016.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-03-26T04:19:48.630 INFO:tasks.workunit.client.0.smithi016.stderr:++ printf %25s api_snapshots 2024-03-26T04:19:48.630 INFO:tasks.workunit.client.0.smithi016.stderr:+ r=' api_snapshots' 2024-03-26T04:19:48.630 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:19:48.630 INFO:tasks.workunit.client.0.smithi016.stderr:+ . /etc/bashrc 2024-03-26T04:19:48.630 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:19:48.630 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -z '' ']' 2024-03-26T04:19:48.630 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:19:48.630 INFO:tasks.workunit.client.0.smithi016.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:19:48.630 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.630 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:19:48.631 INFO:tasks.workunit.client.0.smithi016.stderr:++ shopt -q login_shell 2024-03-26T04:19:48.631 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.631 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:19:48.631 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.631 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:19:48.631 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:19:48.631 INFO:tasks.workunit.client.0.smithi016.stderr:+++ return 2024-03-26T04:19:48.631 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.631 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:19:48.631 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.631 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:19:48.631 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export S_COLORS=auto 2024-03-26T04:19:48.631 INFO:tasks.workunit.client.0.smithi016.stderr:+++ S_COLORS=auto 2024-03-26T04:19:48.631 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.632 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:19:48.632 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.632 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:19:48.632 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.632 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:19:48.632 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:19:48.632 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:19:48.632 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.632 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:19:48.632 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.632 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:19:48.632 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:19:48.632 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.632 INFO:tasks.workunit.client.0.smithi016.stderr:++ awk '{print $1}' 2024-03-26T04:19:48.632 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:19:48.633 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:19:48.633 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:19:48.633 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.633 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:19:48.633 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.633 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:19:48.633 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:19:48.633 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.633 INFO:tasks.workunit.client.0.smithi016.stderr:++ echo api_snapshots 2024-03-26T04:19:48.633 INFO:tasks.workunit.client.0.smithi016.stderr:+ ff=api_snapshots 2024-03-26T04:19:48.633 INFO:tasks.workunit.client.0.smithi016.stdout:test api_snapshots on pid 74439 2024-03-26T04:19:48.635 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74439 2024-03-26T04:19:48.636 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'test api_snapshots on pid 74439' 2024-03-26T04:19:48.636 INFO:tasks.workunit.client.0.smithi016.stderr:+ pids[$f]=74439 2024-03-26T04:19:48.636 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:19:48.636 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:19:48.636 INFO:tasks.workunit.client.0.smithi016.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-03-26T04:19:48.636 INFO:tasks.workunit.client.0.smithi016.stderr:+++ umask 2024-03-26T04:19:48.636 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:19:48.636 INFO:tasks.workunit.client.0.smithi016.stderr:++ SHELL=/bin/bash 2024-03-26T04:19:48.636 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.636 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:19:48.636 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.636 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:19:48.637 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:19:48.637 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.637 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:19:48.637 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.637 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:19:48.637 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.637 INFO:tasks.workunit.client.0.smithi016.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.637 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.637 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:19:48.637 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:19:48.637 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.637 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:19:48.637 INFO:tasks.workunit.client.0.smithi016.stderr:+ . /etc/bashrc 2024-03-26T04:19:48.637 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.638 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -z '' ']' 2024-03-26T04:19:48.638 INFO:tasks.workunit.client.0.smithi016.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:19:48.638 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:19:48.638 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.638 INFO:tasks.workunit.client.0.smithi016.stderr:++ shopt -q login_shell 2024-03-26T04:19:48.638 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:19:48.638 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:19:48.638 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.638 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:19:48.638 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.638 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:19:48.638 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:19:48.638 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.638 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:19:48.639 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.639 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:19:48.639 INFO:tasks.workunit.client.0.smithi016.stderr:+++ umask 2024-03-26T04:19:48.639 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.639 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:19:48.639 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.639 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:19:48.639 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup 2024-03-26T04:19:48.639 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:19:48.639 INFO:tasks.workunit.client.0.smithi016.stderr:++ SHELL=/bin/bash 2024-03-26T04:19:48.639 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.639 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:19:48.639 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.639 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:19:48.640 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:19:48.640 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.640 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:19:48.640 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.640 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:19:48.640 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.640 INFO:tasks.workunit.client.0.smithi016.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.640 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.640 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:19:48.640 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.640 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:19:48.640 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.640 INFO:tasks.workunit.client.0.smithi016.stderr:++ printf %25s api_snapshots_pp 2024-03-26T04:19:48.640 INFO:tasks.workunit.client.0.smithi016.stderr:+ r=' api_snapshots_pp' 2024-03-26T04:19:48.641 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/locale 2024-03-26T04:19:48.641 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:19:48.641 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:19:48.641 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:19:48.641 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.641 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:19:48.641 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.641 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:19:48.641 INFO:tasks.workunit.client.0.smithi016.stderr:++ awk '{print $1}' 2024-03-26T04:19:48.641 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:19:48.641 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.641 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:19:48.641 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.641 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:19:48.641 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.642 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:19:48.642 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.642 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:19:48.642 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup 2024-03-26T04:19:48.642 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/locale 2024-03-26T04:19:48.642 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.642 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:19:48.642 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.642 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:19:48.642 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:19:48.642 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.642 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:19:48.642 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.642 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:19:48.643 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:19:48.643 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:19:48.643 INFO:tasks.workunit.client.0.smithi016.stderr:++ echo api_snapshots_pp 2024-03-26T04:19:48.643 INFO:tasks.workunit.client.0.smithi016.stdout:test api_snapshots_pp on pid 74484 2024-03-26T04:19:48.648 INFO:tasks.workunit.client.0.smithi016.stderr:+ ff=api_snapshots_pp 2024-03-26T04:19:48.648 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74484 2024-03-26T04:19:48.648 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'test api_snapshots_pp on pid 74484' 2024-03-26T04:19:48.648 INFO:tasks.workunit.client.0.smithi016.stderr:+ pids[$f]=74484 2024-03-26T04:19:48.648 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:19:48.648 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:19:48.648 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.648 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:19:48.648 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.648 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:19:48.648 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:19:48.649 INFO:tasks.workunit.client.0.smithi016.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:19:48.649 INFO:tasks.workunit.client.0.smithi016.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:19:48.649 INFO:tasks.workunit.client.0.smithi016.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:19:48.649 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.649 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:19:48.649 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.649 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:19:48.649 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup config 2024-03-26T04:19:48.649 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.649 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.649 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:19:48.649 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.649 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:19:48.649 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.650 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:19:48.650 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:19:48.650 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.650 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:19:48.650 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.650 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:19:48.650 INFO:tasks.workunit.client.0.smithi016.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:19:48.650 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:19:48.650 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:19:48.650 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:19:48.650 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:19:48.650 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.650 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:19:48.651 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.651 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:19:48.651 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:19:48.651 INFO:tasks.workunit.client.0.smithi016.stderr:+++ return 2024-03-26T04:19:48.651 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.651 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:19:48.651 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.651 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:19:48.651 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export S_COLORS=auto 2024-03-26T04:19:48.651 INFO:tasks.workunit.client.0.smithi016.stderr:+++ S_COLORS=auto 2024-03-26T04:19:48.651 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.651 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:19:48.651 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.651 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:19:48.652 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.652 INFO:tasks.workunit.client.0.smithi016.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-03-26T04:19:48.652 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:19:48.652 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:19:48.652 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:19:48.652 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.652 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:19:48.652 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.652 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:19:48.652 INFO:tasks.workunit.client.0.smithi016.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:19:48.652 INFO:tasks.workunit.client.0.smithi016.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:19:48.652 INFO:tasks.workunit.client.0.smithi016.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:19:48.652 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:19:48.652 INFO:tasks.workunit.client.0.smithi016.stderr:+++ return 2024-03-26T04:19:48.653 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.653 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.653 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:19:48.653 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:19:48.653 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.653 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:19:48.653 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.653 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:19:48.653 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export S_COLORS=auto 2024-03-26T04:19:48.653 INFO:tasks.workunit.client.0.smithi016.stderr:+++ S_COLORS=auto 2024-03-26T04:19:48.653 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup config 2024-03-26T04:19:48.653 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.653 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:19:48.654 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.654 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:19:48.654 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.654 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.654 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.654 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:19:48.654 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.654 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:19:48.654 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.654 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:19:48.654 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:19:48.654 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.654 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:19:48.654 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.654 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:19:48.655 INFO:tasks.workunit.client.0.smithi016.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:19:48.655 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:19:48.655 INFO:tasks.workunit.client.0.smithi016.stderr:+ . /etc/bashrc 2024-03-26T04:19:48.655 INFO:tasks.workunit.client.0.smithi016.stderr:++ printf %25s api_stat 2024-03-26T04:19:48.655 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -z '' ']' 2024-03-26T04:19:48.655 INFO:tasks.workunit.client.0.smithi016.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:19:48.655 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.655 INFO:tasks.workunit.client.0.smithi016.stderr:++ shopt -q login_shell 2024-03-26T04:19:48.655 INFO:tasks.workunit.client.0.smithi016.stderr:+ r=' api_stat' 2024-03-26T04:19:48.655 INFO:tasks.workunit.client.0.smithi016.stderr:+++++ readlink /proc/74323/exe 2024-03-26T04:19:48.655 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:19:48.655 INFO:tasks.workunit.client.0.smithi016.stderr:+++ umask 2024-03-26T04:19:48.655 INFO:tasks.workunit.client.0.smithi016.stderr:++++ basename /usr/bin/bash 2024-03-26T04:19:48.655 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:19:48.656 INFO:tasks.workunit.client.0.smithi016.stderr:++ SHELL=/bin/bash 2024-03-26T04:19:48.656 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.656 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:19:48.656 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.656 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:19:48.656 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:19:48.656 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.656 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:19:48.656 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.656 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:19:48.656 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:19:48.656 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.656 INFO:tasks.workunit.client.0.smithi016.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.656 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.656 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:19:48.657 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.657 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:19:48.657 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:19:48.657 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:19:48.657 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.657 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.657 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:19:48.657 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.657 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:19:48.657 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:19:48.657 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.657 INFO:tasks.workunit.client.0.smithi016.stderr:+++++ readlink /proc/74343/exe 2024-03-26T04:19:48.657 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_declare='declare -f' 2024-03-26T04:19:48.658 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_opt=-f 2024-03-26T04:19:48.658 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export which_declare 2024-03-26T04:19:48.658 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export -f which 2024-03-26T04:19:48.658 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset i 2024-03-26T04:19:48.658 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset -f pathmunge 2024-03-26T04:19:48.658 INFO:tasks.workunit.client.0.smithi016.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:19:48.658 INFO:tasks.workunit.client.0.smithi016.stderr:+ export PATH 2024-03-26T04:19:48.658 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:19:48.658 INFO:tasks.workunit.client.0.smithi016.stderr:+ unset rc 2024-03-26T04:19:48.658 INFO:tasks.workunit.client.0.smithi016.stderr:++++ basename /usr/bin/bash 2024-03-26T04:19:48.658 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2024-03-26T04:19:48.658 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_declare='declare -f' 2024-03-26T04:19:48.658 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_opt=-f 2024-03-26T04:19:48.658 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export which_declare 2024-03-26T04:19:48.659 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export -f which 2024-03-26T04:19:48.659 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset i 2024-03-26T04:19:48.659 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset -f pathmunge 2024-03-26T04:19:48.659 INFO:tasks.workunit.client.0.smithi016.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:19:48.659 INFO:tasks.workunit.client.0.smithi016.stderr:+ export PATH 2024-03-26T04:19:48.659 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:19:48.659 INFO:tasks.workunit.client.0.smithi016.stderr:+ unset rc 2024-03-26T04:19:48.659 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:19:48.659 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:19:48.659 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:19:48.659 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.659 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:19:48.659 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.659 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:19:48.660 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:19:48.660 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.660 INFO:tasks.workunit.client.0.smithi016.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:19:48.660 INFO:tasks.workunit.client.0.smithi016.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:19:48.660 INFO:tasks.workunit.client.0.smithi016.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:19:48.660 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.660 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:19:48.660 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.660 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:19:48.660 INFO:tasks.workunit.client.0.smithi016.stderr:+ sed 's/^/ api_misc_pp: /' 2024-03-26T04:19:48.660 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup config 2024-03-26T04:19:48.660 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.660 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.661 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:19:48.661 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.661 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:19:48.661 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.661 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:19:48.661 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:19:48.661 INFO:tasks.workunit.client.0.smithi016.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-03-26T04:19:48.661 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.661 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:19:48.661 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.661 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:19:48.661 INFO:tasks.workunit.client.0.smithi016.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:19:48.661 INFO:tasks.workunit.client.0.smithi016.stderr:+ sed 's/^/ api_misc: /' 2024-03-26T04:19:48.661 INFO:tasks.workunit.client.0.smithi016.stderr:+ tee ceph_test_rados_api_misc.log 2024-03-26T04:19:48.662 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2024-03-26T04:19:48.662 INFO:tasks.workunit.client.0.smithi016.stderr:++ echo api_stat 2024-03-26T04:19:48.662 INFO:tasks.workunit.client.0.smithi016.stderr:+++++ readlink /proc/74360/exe 2024-03-26T04:19:48.662 INFO:tasks.workunit.client.0.smithi016.stderr:++++ basename /usr/bin/bash 2024-03-26T04:19:48.662 INFO:tasks.workunit.client.0.smithi016.stderr:++ awk '{print $1}' 2024-03-26T04:19:48.662 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:19:48.662 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:19:48.662 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:19:48.662 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.662 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:19:48.662 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.662 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:19:48.662 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:19:48.662 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.662 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:19:48.663 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.663 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:19:48.663 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.663 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:19:48.663 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.663 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:19:48.663 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup 2024-03-26T04:19:48.663 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_declare='declare -f' 2024-03-26T04:19:48.663 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_opt=-f 2024-03-26T04:19:48.663 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export which_declare 2024-03-26T04:19:48.663 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export -f which 2024-03-26T04:19:48.663 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset i 2024-03-26T04:19:48.663 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset -f pathmunge 2024-03-26T04:19:48.663 INFO:tasks.workunit.client.0.smithi016.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:19:48.664 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/locale 2024-03-26T04:19:48.664 INFO:tasks.workunit.client.0.smithi016.stderr:+ export PATH 2024-03-26T04:19:48.664 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:19:48.664 INFO:tasks.workunit.client.0.smithi016.stderr:+ unset rc 2024-03-26T04:19:48.664 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:19:48.664 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:19:48.664 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:19:48.664 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.664 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:19:48.664 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.664 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:19:48.664 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:19:48.664 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.665 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:19:48.665 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.665 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:19:48.665 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.665 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:19:48.665 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.665 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:19:48.665 INFO:tasks.workunit.client.0.smithi016.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-03-26T04:19:48.665 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup 2024-03-26T04:19:48.665 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.665 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:19:48.665 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.665 INFO:tasks.workunit.client.0.smithi016.stderr:+ ff=api_stat 2024-03-26T04:19:48.665 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:19:48.666 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:19:48.666 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:19:48.666 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74538 2024-03-26T04:19:48.666 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'test api_stat on pid 74538' 2024-03-26T04:19:48.666 INFO:tasks.workunit.client.0.smithi016.stdout:test api_stat on pid 74538 2024-03-26T04:19:48.668 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:19:48.669 INFO:tasks.workunit.client.0.smithi016.stderr:+ pids[$f]=74538 2024-03-26T04:19:48.669 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:19:48.669 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:19:48.669 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:19:48.669 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.669 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:19:48.669 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.669 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:19:48.669 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:19:48.669 INFO:tasks.workunit.client.0.smithi016.stderr:+++ return 2024-03-26T04:19:48.669 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.669 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:19:48.669 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.670 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:19:48.670 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export S_COLORS=auto 2024-03-26T04:19:48.670 INFO:tasks.workunit.client.0.smithi016.stderr:+++ S_COLORS=auto 2024-03-26T04:19:48.670 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.670 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:19:48.670 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.670 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:19:48.670 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:19:48.670 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.670 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/locale 2024-03-26T04:19:48.670 INFO:tasks.workunit.client.0.smithi016.stderr:++ printf %25s api_stat_pp 2024-03-26T04:19:48.670 INFO:tasks.workunit.client.0.smithi016.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-03-26T04:19:48.670 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.671 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:19:48.671 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.671 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:19:48.671 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:19:48.671 INFO:tasks.workunit.client.0.smithi016.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:19:48.671 INFO:tasks.workunit.client.0.smithi016.stderr:+ r=' api_stat_pp' 2024-03-26T04:19:48.671 INFO:tasks.workunit.client.0.smithi016.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:19:48.671 INFO:tasks.workunit.client.0.smithi016.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:19:48.671 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.671 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:19:48.671 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.671 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:19:48.671 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup config 2024-03-26T04:19:48.671 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.671 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:19:48.672 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.672 INFO:tasks.workunit.client.0.smithi016.stderr:+ . /etc/bashrc 2024-03-26T04:19:48.672 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:19:48.672 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:19:48.672 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.672 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:19:48.672 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.672 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:19:48.672 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:19:48.672 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.672 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:19:48.672 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.672 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:19:48.672 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -z '' ']' 2024-03-26T04:19:48.673 INFO:tasks.workunit.client.0.smithi016.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:19:48.673 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.673 INFO:tasks.workunit.client.0.smithi016.stderr:++ shopt -q login_shell 2024-03-26T04:19:48.673 INFO:tasks.workunit.client.0.smithi016.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:19:48.673 INFO:tasks.workunit.client.0.smithi016.stderr:++ echo api_stat_pp 2024-03-26T04:19:48.673 INFO:tasks.workunit.client.0.smithi016.stderr:+++ umask 2024-03-26T04:19:48.673 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:19:48.673 INFO:tasks.workunit.client.0.smithi016.stderr:++ SHELL=/bin/bash 2024-03-26T04:19:48.673 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2024-03-26T04:19:48.673 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.673 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:19:48.673 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.673 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:19:48.673 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:19:48.673 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.674 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:19:48.674 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.674 INFO:tasks.workunit.client.0.smithi016.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:19:48.674 INFO:tasks.workunit.client.0.smithi016.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:19:48.674 INFO:tasks.workunit.client.0.smithi016.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:19:48.674 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:19:48.674 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.674 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.674 INFO:tasks.workunit.client.0.smithi016.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.674 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:19:48.674 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.674 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:19:48.674 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.674 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.675 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:19:48.675 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:19:48.675 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup config 2024-03-26T04:19:48.675 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.675 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.675 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.675 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:19:48.675 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.675 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:19:48.675 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.675 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:19:48.675 INFO:tasks.workunit.client.0.smithi016.stderr:+ sed 's/^/ api_tier_pp: /' 2024-03-26T04:19:48.675 INFO:tasks.workunit.client.0.smithi016.stderr:++ awk '{print $1}' 2024-03-26T04:19:48.676 INFO:tasks.workunit.client.0.smithi016.stdout:test api_stat_pp on pid 74575 2024-03-26T04:19:48.681 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:19:48.681 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.681 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:19:48.681 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.681 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:19:48.681 INFO:tasks.workunit.client.0.smithi016.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:19:48.681 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:19:48.681 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:19:48.682 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:19:48.682 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.682 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:19:48.682 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.682 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:19:48.682 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:19:48.682 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.682 INFO:tasks.workunit.client.0.smithi016.stderr:+++++ readlink /proc/74439/exe 2024-03-26T04:19:48.682 INFO:tasks.workunit.client.0.smithi016.stderr:+ ff=api_stat_pp 2024-03-26T04:19:48.682 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74575 2024-03-26T04:19:48.682 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'test api_stat_pp on pid 74575' 2024-03-26T04:19:48.682 INFO:tasks.workunit.client.0.smithi016.stderr:+ pids[$f]=74575 2024-03-26T04:19:48.683 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:19:48.683 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:19:48.683 INFO:tasks.workunit.client.0.smithi016.stderr:++++ basename /usr/bin/bash 2024-03-26T04:19:48.683 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_declare='declare -f' 2024-03-26T04:19:48.683 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_opt=-f 2024-03-26T04:19:48.683 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export which_declare 2024-03-26T04:19:48.683 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export -f which 2024-03-26T04:19:48.683 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset i 2024-03-26T04:19:48.683 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset -f pathmunge 2024-03-26T04:19:48.683 INFO:tasks.workunit.client.0.smithi016.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:19:48.683 INFO:tasks.workunit.client.0.smithi016.stderr:+ export PATH 2024-03-26T04:19:48.683 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:19:48.683 INFO:tasks.workunit.client.0.smithi016.stderr:+ unset rc 2024-03-26T04:19:48.683 INFO:tasks.workunit.client.0.smithi016.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-03-26T04:19:48.684 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:19:48.684 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:19:48.684 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2024-03-26T04:19:48.684 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:19:48.684 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.684 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:19:48.684 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.684 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:19:48.684 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:19:48.684 INFO:tasks.workunit.client.0.smithi016.stderr:+++ return 2024-03-26T04:19:48.684 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.684 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:19:48.684 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.684 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:19:48.685 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export S_COLORS=auto 2024-03-26T04:19:48.685 INFO:tasks.workunit.client.0.smithi016.stderr:+++ S_COLORS=auto 2024-03-26T04:19:48.685 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.685 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:19:48.685 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.685 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:19:48.685 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.685 INFO:tasks.workunit.client.0.smithi016.stderr:+ sed 's/^/ api_snapshots: /' 2024-03-26T04:19:48.685 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:19:48.685 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:19:48.685 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:19:48.685 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.685 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:19:48.685 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.685 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:19:48.686 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:19:48.686 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.686 INFO:tasks.workunit.client.0.smithi016.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-03-26T04:19:48.686 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:19:48.686 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:19:48.686 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:19:48.686 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.686 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:19:48.686 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.686 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:19:48.687 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:19:48.687 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.687 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:19:48.687 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.687 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:19:48.687 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.687 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:19:48.687 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.687 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:19:48.687 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup 2024-03-26T04:19:48.687 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/locale 2024-03-26T04:19:48.687 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:19:48.687 INFO:tasks.workunit.client.0.smithi016.stderr:+ . /etc/bashrc 2024-03-26T04:19:48.687 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -z '' ']' 2024-03-26T04:19:48.688 INFO:tasks.workunit.client.0.smithi016.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:19:48.688 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.688 INFO:tasks.workunit.client.0.smithi016.stderr:++ shopt -q login_shell 2024-03-26T04:19:48.688 INFO:tasks.workunit.client.0.smithi016.stderr:+++ umask 2024-03-26T04:19:48.688 INFO:tasks.workunit.client.0.smithi016.stderr:+++++ readlink /proc/74399/exe 2024-03-26T04:19:48.688 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:19:48.688 INFO:tasks.workunit.client.0.smithi016.stderr:++ SHELL=/bin/bash 2024-03-26T04:19:48.688 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.688 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:19:48.688 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.688 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:19:48.688 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.688 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:19:48.688 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.689 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:19:48.689 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:19:48.689 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:19:48.689 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.689 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:19:48.689 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.689 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:19:48.689 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.689 INFO:tasks.workunit.client.0.smithi016.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.689 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.689 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:19:48.689 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.689 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:19:48.689 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.689 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:19:48.690 INFO:tasks.workunit.client.0.smithi016.stderr:++++ basename /usr/bin/bash 2024-03-26T04:19:48.690 INFO:tasks.workunit.client.0.smithi016.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:19:48.690 INFO:tasks.workunit.client.0.smithi016.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:19:48.690 INFO:tasks.workunit.client.0.smithi016.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:19:48.690 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.690 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:19:48.690 INFO:tasks.workunit.client.0.smithi016.stderr:++ printf %25s api_watch_notify 2024-03-26T04:19:48.690 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.690 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:19:48.690 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup config 2024-03-26T04:19:48.690 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.690 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.690 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:19:48.690 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.691 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:19:48.691 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.691 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:19:48.691 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:19:48.691 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.691 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:19:48.691 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.691 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:19:48.691 INFO:tasks.workunit.client.0.smithi016.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:19:48.691 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_declare='declare -f' 2024-03-26T04:19:48.691 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_opt=-f 2024-03-26T04:19:48.691 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export which_declare 2024-03-26T04:19:48.691 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export -f which 2024-03-26T04:19:48.691 INFO:tasks.workunit.client.0.smithi016.stderr:+ r=' api_watch_notify' 2024-03-26T04:19:48.691 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset i 2024-03-26T04:19:48.692 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset -f pathmunge 2024-03-26T04:19:48.692 INFO:tasks.workunit.client.0.smithi016.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:19:48.692 INFO:tasks.workunit.client.0.smithi016.stderr:+ export PATH 2024-03-26T04:19:48.692 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:19:48.692 INFO:tasks.workunit.client.0.smithi016.stderr:+ unset rc 2024-03-26T04:19:48.692 INFO:tasks.workunit.client.0.smithi016.stderr:+++++ readlink /proc/74484/exe 2024-03-26T04:19:48.692 INFO:tasks.workunit.client.0.smithi016.stderr:++++ basename /usr/bin/bash 2024-03-26T04:19:48.692 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2024-03-26T04:19:48.692 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:19:48.692 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:19:48.692 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:19:48.692 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.692 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:19:48.692 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.692 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:19:48.693 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:19:48.693 INFO:tasks.workunit.client.0.smithi016.stderr:+++ return 2024-03-26T04:19:48.693 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.693 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:19:48.693 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.693 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:19:48.693 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export S_COLORS=auto 2024-03-26T04:19:48.693 INFO:tasks.workunit.client.0.smithi016.stderr:+++ S_COLORS=auto 2024-03-26T04:19:48.693 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.693 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:19:48.693 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.693 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:19:48.693 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_declare='declare -f' 2024-03-26T04:19:48.693 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_opt=-f 2024-03-26T04:19:48.694 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.694 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export which_declare 2024-03-26T04:19:48.694 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export -f which 2024-03-26T04:19:48.694 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset i 2024-03-26T04:19:48.694 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset -f pathmunge 2024-03-26T04:19:48.694 INFO:tasks.workunit.client.0.smithi016.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:19:48.694 INFO:tasks.workunit.client.0.smithi016.stderr:+ export PATH 2024-03-26T04:19:48.694 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:19:48.694 INFO:tasks.workunit.client.0.smithi016.stderr:+ unset rc 2024-03-26T04:19:48.694 INFO:tasks.workunit.client.0.smithi016.stderr:+ tee ceph_test_rados_api_pool.log 2024-03-26T04:19:48.694 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:19:48.694 INFO:tasks.workunit.client.0.smithi016.stderr:++ echo api_watch_notify 2024-03-26T04:19:48.694 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:19:48.694 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:19:48.694 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.695 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:19:48.695 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.695 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:19:48.695 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:19:48.695 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.695 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:19:48.695 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.695 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:19:48.695 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.695 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:19:48.695 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.695 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:19:48.695 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup 2024-03-26T04:19:48.695 INFO:tasks.workunit.client.0.smithi016.stderr:++ awk '{print $1}' 2024-03-26T04:19:48.696 INFO:tasks.workunit.client.0.smithi016.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-03-26T04:19:48.696 INFO:tasks.workunit.client.0.smithi016.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-03-26T04:19:48.696 INFO:tasks.workunit.client.0.smithi016.stderr:+ sed 's/^/ api_pool: /' 2024-03-26T04:19:48.696 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2024-03-26T04:19:48.696 INFO:tasks.workunit.client.0.smithi016.stderr:+ ff=api_watch_notify 2024-03-26T04:19:48.696 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74624 2024-03-26T04:19:48.696 INFO:tasks.workunit.client.0.smithi016.stdout:test api_watch_notify on pid 74624 2024-03-26T04:19:48.697 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'test api_watch_notify on pid 74624' 2024-03-26T04:19:48.697 INFO:tasks.workunit.client.0.smithi016.stderr:+ pids[$f]=74624 2024-03-26T04:19:48.697 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:19:48.697 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:19:48.697 INFO:tasks.workunit.client.0.smithi016.stderr:++ printf %25s api_watch_notify_pp 2024-03-26T04:19:48.697 INFO:tasks.workunit.client.0.smithi016.stderr:+ r=' api_watch_notify_pp' 2024-03-26T04:19:48.697 INFO:tasks.workunit.client.0.smithi016.stderr:++ echo api_watch_notify_pp 2024-03-26T04:19:48.698 INFO:tasks.workunit.client.0.smithi016.stderr:++ awk '{print $1}' 2024-03-26T04:19:48.698 INFO:tasks.workunit.client.0.smithi016.stdout:test api_watch_notify_pp on pid 74630 2024-03-26T04:19:48.699 INFO:tasks.workunit.client.0.smithi016.stderr:+ ff=api_watch_notify_pp 2024-03-26T04:19:48.700 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74630 2024-03-26T04:19:48.700 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'test api_watch_notify_pp on pid 74630' 2024-03-26T04:19:48.700 INFO:tasks.workunit.client.0.smithi016.stderr:+ pids[$f]=74630 2024-03-26T04:19:48.700 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:19:48.700 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:19:48.700 INFO:tasks.workunit.client.0.smithi016.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-03-26T04:19:48.700 INFO:tasks.workunit.client.0.smithi016.stderr:++ printf %25s api_cmd 2024-03-26T04:19:48.700 INFO:tasks.workunit.client.0.smithi016.stderr:+ r=' api_cmd' 2024-03-26T04:19:48.700 INFO:tasks.workunit.client.0.smithi016.stderr:++ echo api_cmd 2024-03-26T04:19:48.700 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:19:48.700 INFO:tasks.workunit.client.0.smithi016.stderr:+ . /etc/bashrc 2024-03-26T04:19:48.700 INFO:tasks.workunit.client.0.smithi016.stderr:++ awk '{print $1}' 2024-03-26T04:19:48.700 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -z '' ']' 2024-03-26T04:19:48.701 INFO:tasks.workunit.client.0.smithi016.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:19:48.701 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.701 INFO:tasks.workunit.client.0.smithi016.stderr:++ shopt -q login_shell 2024-03-26T04:19:48.701 INFO:tasks.workunit.client.0.smithi016.stderr:+++ umask 2024-03-26T04:19:48.701 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:19:48.701 INFO:tasks.workunit.client.0.smithi016.stderr:++ SHELL=/bin/bash 2024-03-26T04:19:48.701 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.701 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:19:48.701 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.701 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:19:48.701 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:19:48.701 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.701 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:19:48.701 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.702 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:19:48.702 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.702 INFO:tasks.workunit.client.0.smithi016.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.702 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.702 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:19:48.702 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.702 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:19:48.702 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.702 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/locale 2024-03-26T04:19:48.702 INFO:tasks.workunit.client.0.smithi016.stdout:test api_cmd on pid 74638 2024-03-26T04:19:48.707 INFO:tasks.workunit.client.0.smithi016.stderr:+ ff=api_cmd 2024-03-26T04:19:48.707 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74638 2024-03-26T04:19:48.707 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'test api_cmd on pid 74638' 2024-03-26T04:19:48.707 INFO:tasks.workunit.client.0.smithi016.stderr:+ pids[$f]=74638 2024-03-26T04:19:48.707 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:19:48.707 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:19:48.707 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:19:48.707 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:19:48.708 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:19:48.708 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.708 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:19:48.708 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.708 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:19:48.708 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:19:48.708 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.708 INFO:tasks.workunit.client.0.smithi016.stderr:++ printf %25s api_cmd_pp 2024-03-26T04:19:48.708 INFO:tasks.workunit.client.0.smithi016.stderr:+ r=' api_cmd_pp' 2024-03-26T04:19:48.708 INFO:tasks.workunit.client.0.smithi016.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-03-26T04:19:48.708 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.708 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:19:48.708 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.708 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:19:48.708 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:19:48.709 INFO:tasks.workunit.client.0.smithi016.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-03-26T04:19:48.709 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:19:48.709 INFO:tasks.workunit.client.0.smithi016.stderr:+ . /etc/bashrc 2024-03-26T04:19:48.709 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -z '' ']' 2024-03-26T04:19:48.709 INFO:tasks.workunit.client.0.smithi016.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:19:48.709 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.709 INFO:tasks.workunit.client.0.smithi016.stderr:++ shopt -q login_shell 2024-03-26T04:19:48.709 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:19:48.709 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:19:48.709 INFO:tasks.workunit.client.0.smithi016.stderr:+ . /etc/bashrc 2024-03-26T04:19:48.709 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -z '' ']' 2024-03-26T04:19:48.709 INFO:tasks.workunit.client.0.smithi016.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:19:48.709 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.709 INFO:tasks.workunit.client.0.smithi016.stderr:++ shopt -q login_shell 2024-03-26T04:19:48.710 INFO:tasks.workunit.client.0.smithi016.stderr:++ echo api_cmd_pp 2024-03-26T04:19:48.710 INFO:tasks.workunit.client.0.smithi016.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:19:48.710 INFO:tasks.workunit.client.0.smithi016.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:19:48.710 INFO:tasks.workunit.client.0.smithi016.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:19:48.710 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.710 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:19:48.710 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.710 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:19:48.710 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup config 2024-03-26T04:19:48.710 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.710 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.710 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:19:48.710 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.711 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:19:48.711 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.711 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:19:48.711 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:19:48.711 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.711 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:19:48.711 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.711 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:19:48.711 INFO:tasks.workunit.client.0.smithi016.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:19:48.711 INFO:tasks.workunit.client.0.smithi016.stderr:+++ umask 2024-03-26T04:19:48.711 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:19:48.711 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:19:48.711 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:19:48.711 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.712 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:19:48.712 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.712 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:19:48.712 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:19:48.712 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.712 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:19:48.712 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.712 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:19:48.712 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.712 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:19:48.712 INFO:tasks.workunit.client.0.smithi016.stderr:+++++ readlink /proc/74538/exe 2024-03-26T04:19:48.712 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.712 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:19:48.712 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup 2024-03-26T04:19:48.713 INFO:tasks.workunit.client.0.smithi016.stderr:+++ umask 2024-03-26T04:19:48.713 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:19:48.713 INFO:tasks.workunit.client.0.smithi016.stderr:++ SHELL=/bin/bash 2024-03-26T04:19:48.713 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.713 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:19:48.713 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.713 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:19:48.713 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:19:48.713 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:19:48.713 INFO:tasks.workunit.client.0.smithi016.stderr:++ SHELL=/bin/bash 2024-03-26T04:19:48.713 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.713 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:19:48.713 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.713 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:19:48.713 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.714 INFO:tasks.workunit.client.0.smithi016.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.714 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.714 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.714 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:19:48.714 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.714 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:19:48.714 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:19:48.714 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.714 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:19:48.714 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.714 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:19:48.714 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.714 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:19:48.714 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.715 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:19:48.715 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.715 INFO:tasks.workunit.client.0.smithi016.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.715 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.715 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:19:48.715 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.715 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:19:48.715 INFO:tasks.workunit.client.0.smithi016.stderr:++++ basename /usr/bin/bash 2024-03-26T04:19:48.715 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.715 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/locale 2024-03-26T04:19:48.715 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_declare='declare -f' 2024-03-26T04:19:48.715 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_opt=-f 2024-03-26T04:19:48.715 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export which_declare 2024-03-26T04:19:48.715 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export -f which 2024-03-26T04:19:48.715 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset i 2024-03-26T04:19:48.716 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset -f pathmunge 2024-03-26T04:19:48.716 INFO:tasks.workunit.client.0.smithi016.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:19:48.716 INFO:tasks.workunit.client.0.smithi016.stderr:+ export PATH 2024-03-26T04:19:48.716 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:19:48.716 INFO:tasks.workunit.client.0.smithi016.stderr:+ unset rc 2024-03-26T04:19:48.716 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:19:48.716 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:19:48.716 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:19:48.716 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.716 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:19:48.716 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.716 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:19:48.716 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:19:48.716 INFO:tasks.workunit.client.0.smithi016.stderr:+++ return 2024-03-26T04:19:48.717 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.717 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:19:48.717 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.717 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:19:48.717 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export S_COLORS=auto 2024-03-26T04:19:48.717 INFO:tasks.workunit.client.0.smithi016.stderr:+++ S_COLORS=auto 2024-03-26T04:19:48.717 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.717 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:19:48.717 INFO:tasks.workunit.client.0.smithi016.stderr:+ sed 's/^/ api_stat: /' 2024-03-26T04:19:48.717 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.717 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:19:48.717 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.717 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2024-03-26T04:19:48.717 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.717 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:19:48.718 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.718 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:19:48.718 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:19:48.718 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:19:48.718 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:19:48.718 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:19:48.718 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:19:48.718 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.718 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:19:48.718 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.718 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:19:48.718 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:19:48.718 INFO:tasks.workunit.client.0.smithi016.stderr:+++ return 2024-03-26T04:19:48.718 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.719 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:19:48.719 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.719 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:19:48.719 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export S_COLORS=auto 2024-03-26T04:19:48.719 INFO:tasks.workunit.client.0.smithi016.stderr:+++ S_COLORS=auto 2024-03-26T04:19:48.719 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.719 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:19:48.719 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.719 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:19:48.719 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.719 INFO:tasks.workunit.client.0.smithi016.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:19:48.719 INFO:tasks.workunit.client.0.smithi016.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:19:48.719 INFO:tasks.workunit.client.0.smithi016.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:19:48.719 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.719 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:19:48.720 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.720 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:19:48.720 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup config 2024-03-26T04:19:48.720 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.720 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.720 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:19:48.720 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.720 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:19:48.720 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.720 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:19:48.720 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:19:48.720 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.720 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:19:48.720 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.720 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:19:48.721 INFO:tasks.workunit.client.0.smithi016.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:19:48.721 INFO:tasks.workunit.client.0.smithi016.stderr:+++++ readlink /proc/74575/exe 2024-03-26T04:19:48.721 INFO:tasks.workunit.client.0.smithi016.stderr:++++ basename /usr/bin/bash 2024-03-26T04:19:48.721 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_declare='declare -f' 2024-03-26T04:19:48.721 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_opt=-f 2024-03-26T04:19:48.721 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export which_declare 2024-03-26T04:19:48.721 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export -f which 2024-03-26T04:19:48.721 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset i 2024-03-26T04:19:48.721 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset -f pathmunge 2024-03-26T04:19:48.721 INFO:tasks.workunit.client.0.smithi016.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:19:48.721 INFO:tasks.workunit.client.0.smithi016.stderr:+ tee ceph_test_rados_api_stat.log 2024-03-26T04:19:48.721 INFO:tasks.workunit.client.0.smithi016.stderr:+ export PATH 2024-03-26T04:19:48.721 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:19:48.721 INFO:tasks.workunit.client.0.smithi016.stderr:+ unset rc 2024-03-26T04:19:48.722 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:19:48.722 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:19:48.722 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:19:48.722 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.722 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:19:48.722 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.722 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:19:48.722 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:19:48.722 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:19:48.722 INFO:tasks.workunit.client.0.smithi016.stderr:+++ return 2024-03-26T04:19:48.722 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.722 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:19:48.722 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:19:48.722 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.722 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:19:48.723 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:19:48.723 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export S_COLORS=auto 2024-03-26T04:19:48.723 INFO:tasks.workunit.client.0.smithi016.stderr:+++ S_COLORS=auto 2024-03-26T04:19:48.723 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.723 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.723 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:19:48.723 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:19:48.723 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.723 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.723 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:19:48.723 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:19:48.723 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.723 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:19:48.723 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.723 INFO:tasks.workunit.client.0.smithi016.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-03-26T04:19:48.724 INFO:tasks.workunit.client.0.smithi016.stderr:+ sed 's/^/ api_stat_pp: /' 2024-03-26T04:19:48.724 INFO:tasks.workunit.client.0.smithi016.stderr:++ awk '{print $1}' 2024-03-26T04:19:48.724 INFO:tasks.workunit.client.0.smithi016.stdout:test api_cmd_pp on pid 74709 2024-03-26T04:19:48.726 INFO:tasks.workunit.client.0.smithi016.stderr:+ ff=api_cmd_pp 2024-03-26T04:19:48.726 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74709 2024-03-26T04:19:48.726 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'test api_cmd_pp on pid 74709' 2024-03-26T04:19:48.726 INFO:tasks.workunit.client.0.smithi016.stderr:+ pids[$f]=74709 2024-03-26T04:19:48.727 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:19:48.727 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:19:48.727 INFO:tasks.workunit.client.0.smithi016.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-03-26T04:19:48.727 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2024-03-26T04:19:48.727 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:19:48.727 INFO:tasks.workunit.client.0.smithi016.stderr:+ . /etc/bashrc 2024-03-26T04:19:48.727 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -z '' ']' 2024-03-26T04:19:48.727 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:19:48.727 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:19:48.727 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:19:48.727 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.727 INFO:tasks.workunit.client.0.smithi016.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:19:48.727 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:19:48.727 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.728 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.728 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:19:48.728 INFO:tasks.workunit.client.0.smithi016.stderr:++ shopt -q login_shell 2024-03-26T04:19:48.728 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:19:48.728 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.728 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:19:48.728 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:19:48.728 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:19:48.728 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.728 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:19:48.728 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.728 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:19:48.728 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:19:48.728 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.728 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:19:48.729 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.729 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:19:48.729 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.729 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:19:48.729 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.729 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:19:48.729 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup 2024-03-26T04:19:48.729 INFO:tasks.workunit.client.0.smithi016.stderr:++ printf %25s api_service 2024-03-26T04:19:48.729 INFO:tasks.workunit.client.0.smithi016.stderr:+ r=' api_service' 2024-03-26T04:19:48.729 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/locale 2024-03-26T04:19:48.729 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:19:48.729 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:19:48.729 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:19:48.729 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.729 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:19:48.730 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.730 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:19:48.730 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:19:48.730 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.730 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:19:48.730 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.730 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:19:48.730 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.730 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:19:48.730 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.730 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:19:48.730 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup 2024-03-26T04:19:48.731 INFO:tasks.workunit.client.0.smithi016.stderr:++ echo api_service 2024-03-26T04:19:48.731 INFO:tasks.workunit.client.0.smithi016.stderr:+++ umask 2024-03-26T04:19:48.731 INFO:tasks.workunit.client.0.smithi016.stderr:++ awk '{print $1}' 2024-03-26T04:19:48.731 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/locale 2024-03-26T04:19:48.731 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:19:48.731 INFO:tasks.workunit.client.0.smithi016.stderr:++ SHELL=/bin/bash 2024-03-26T04:19:48.731 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.731 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:19:48.731 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.731 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:19:48.732 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:19:48.732 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.732 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:19:48.732 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.732 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:19:48.732 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.732 INFO:tasks.workunit.client.0.smithi016.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.732 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.732 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:19:48.732 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.732 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:19:48.732 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.732 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.732 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:19:48.733 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.733 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:19:48.733 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:19:48.733 INFO:tasks.workunit.client.0.smithi016.stderr:+ ff=api_service 2024-03-26T04:19:48.733 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74743 2024-03-26T04:19:48.733 INFO:tasks.workunit.client.0.smithi016.stdout:test api_service on pid 74743 2024-03-26T04:19:48.737 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'test api_service on pid 74743' 2024-03-26T04:19:48.737 INFO:tasks.workunit.client.0.smithi016.stderr:+ pids[$f]=74743 2024-03-26T04:19:48.737 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:19:48.737 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:19:48.737 INFO:tasks.workunit.client.0.smithi016.stderr:++ printf %25s api_service_pp 2024-03-26T04:19:48.737 INFO:tasks.workunit.client.0.smithi016.stderr:+ r=' api_service_pp' 2024-03-26T04:19:48.737 INFO:tasks.workunit.client.0.smithi016.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-03-26T04:19:48.737 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:19:48.737 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:19:48.737 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:19:48.737 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.737 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:19:48.737 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.738 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:19:48.738 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:19:48.738 INFO:tasks.workunit.client.0.smithi016.stderr:+++ return 2024-03-26T04:19:48.738 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.738 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:19:48.738 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.738 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:19:48.738 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export S_COLORS=auto 2024-03-26T04:19:48.738 INFO:tasks.workunit.client.0.smithi016.stderr:+++ S_COLORS=auto 2024-03-26T04:19:48.738 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.738 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:19:48.738 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.738 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:19:48.739 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.739 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:19:48.739 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:19:48.739 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:19:48.739 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.739 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:19:48.739 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.739 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:19:48.739 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:19:48.739 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.739 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:19:48.739 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:19:48.739 INFO:tasks.workunit.client.0.smithi016.stderr:+ . /etc/bashrc 2024-03-26T04:19:48.739 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -z '' ']' 2024-03-26T04:19:48.739 INFO:tasks.workunit.client.0.smithi016.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:19:48.740 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.740 INFO:tasks.workunit.client.0.smithi016.stderr:++ shopt -q login_shell 2024-03-26T04:19:48.740 INFO:tasks.workunit.client.0.smithi016.stderr:++ echo api_service_pp 2024-03-26T04:19:48.740 INFO:tasks.workunit.client.0.smithi016.stderr:++ awk '{print $1}' 2024-03-26T04:19:48.740 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.740 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:19:48.740 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.740 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:19:48.740 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:19:48.740 INFO:tasks.workunit.client.0.smithi016.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:19:48.740 INFO:tasks.workunit.client.0.smithi016.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:19:48.740 INFO:tasks.workunit.client.0.smithi016.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:19:48.740 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.740 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:19:48.741 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.741 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:19:48.741 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup config 2024-03-26T04:19:48.741 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.741 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.741 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:19:48.741 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.741 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:19:48.741 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.741 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:19:48.741 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:19:48.741 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.741 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:19:48.741 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.741 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:19:48.742 INFO:tasks.workunit.client.0.smithi016.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:19:48.742 INFO:tasks.workunit.client.0.smithi016.stderr:+++ umask 2024-03-26T04:19:48.742 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:19:48.742 INFO:tasks.workunit.client.0.smithi016.stderr:++ SHELL=/bin/bash 2024-03-26T04:19:48.742 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.742 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:19:48.742 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.742 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:19:48.742 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:19:48.742 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.742 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:19:48.742 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.742 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:19:48.742 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.743 INFO:tasks.workunit.client.0.smithi016.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.743 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.743 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:19:48.743 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.743 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:19:48.743 INFO:tasks.workunit.client.0.smithi016.stdout:test api_service_pp on pid 74770 2024-03-26T04:19:48.747 INFO:tasks.workunit.client.0.smithi016.stderr:+ ff=api_service_pp 2024-03-26T04:19:48.747 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.747 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74770 2024-03-26T04:19:48.747 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'test api_service_pp on pid 74770' 2024-03-26T04:19:48.747 INFO:tasks.workunit.client.0.smithi016.stderr:+ pids[$f]=74770 2024-03-26T04:19:48.748 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:19:48.748 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:19:48.748 INFO:tasks.workunit.client.0.smithi016.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-03-26T04:19:48.748 INFO:tasks.workunit.client.0.smithi016.stderr:+++++ readlink /proc/74630/exe 2024-03-26T04:19:48.748 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:19:48.748 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:19:48.748 INFO:tasks.workunit.client.0.smithi016.stderr:+ . /etc/bashrc 2024-03-26T04:19:48.748 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -z '' ']' 2024-03-26T04:19:48.748 INFO:tasks.workunit.client.0.smithi016.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:19:48.748 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.748 INFO:tasks.workunit.client.0.smithi016.stderr:++ shopt -q login_shell 2024-03-26T04:19:48.748 INFO:tasks.workunit.client.0.smithi016.stderr:+++ umask 2024-03-26T04:19:48.748 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:19:48.749 INFO:tasks.workunit.client.0.smithi016.stderr:++ SHELL=/bin/bash 2024-03-26T04:19:48.749 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.749 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:19:48.749 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.749 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:19:48.749 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:19:48.749 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:19:48.749 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:19:48.749 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.749 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:19:48.749 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:19:48.749 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.749 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:19:48.749 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.750 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:19:48.750 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.750 INFO:tasks.workunit.client.0.smithi016.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.750 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.750 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.750 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:19:48.750 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:19:48.750 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.750 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:19:48.750 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:19:48.750 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.750 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.750 INFO:tasks.workunit.client.0.smithi016.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:19:48.751 INFO:tasks.workunit.client.0.smithi016.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:19:48.751 INFO:tasks.workunit.client.0.smithi016.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:19:48.751 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.751 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:19:48.751 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.751 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:19:48.751 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup config 2024-03-26T04:19:48.751 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.751 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.751 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:19:48.751 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.751 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:19:48.751 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.752 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:19:48.752 INFO:tasks.workunit.client.0.smithi016.stderr:++++ basename /usr/bin/bash 2024-03-26T04:19:48.752 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:19:48.752 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.752 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:19:48.752 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.752 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:19:48.752 INFO:tasks.workunit.client.0.smithi016.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:19:48.752 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_declare='declare -f' 2024-03-26T04:19:48.752 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_opt=-f 2024-03-26T04:19:48.752 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export which_declare 2024-03-26T04:19:48.752 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export -f which 2024-03-26T04:19:48.752 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset i 2024-03-26T04:19:48.753 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset -f pathmunge 2024-03-26T04:19:48.753 INFO:tasks.workunit.client.0.smithi016.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:19:48.753 INFO:tasks.workunit.client.0.smithi016.stderr:+ export PATH 2024-03-26T04:19:48.753 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:19:48.753 INFO:tasks.workunit.client.0.smithi016.stderr:+ unset rc 2024-03-26T04:19:48.753 INFO:tasks.workunit.client.0.smithi016.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-03-26T04:19:48.753 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2024-03-26T04:19:48.753 INFO:tasks.workunit.client.0.smithi016.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-03-26T04:19:48.753 INFO:tasks.workunit.client.0.smithi016.stderr:++ printf %25s api_c_write_operations 2024-03-26T04:19:48.753 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:19:48.753 INFO:tasks.workunit.client.0.smithi016.stderr:+ r=' api_c_write_operations' 2024-03-26T04:19:48.753 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:19:48.753 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:19:48.753 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.754 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:19:48.754 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.754 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:19:48.754 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:19:48.754 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.754 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:19:48.754 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.754 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:19:48.754 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.754 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:19:48.754 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.754 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:19:48.754 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup 2024-03-26T04:19:48.754 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/locale 2024-03-26T04:19:48.755 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:19:48.755 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:19:48.755 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:19:48.755 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.755 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:19:48.755 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.755 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:19:48.755 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:19:48.755 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.755 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:19:48.755 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.755 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:19:48.755 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.755 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:19:48.756 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.756 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:19:48.756 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup 2024-03-26T04:19:48.756 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:19:48.756 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:19:48.756 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:19:48.756 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.756 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:19:48.756 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.756 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:19:48.756 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:19:48.756 INFO:tasks.workunit.client.0.smithi016.stderr:+++ return 2024-03-26T04:19:48.756 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.756 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:19:48.756 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.757 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:19:48.757 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export S_COLORS=auto 2024-03-26T04:19:48.757 INFO:tasks.workunit.client.0.smithi016.stderr:+++ S_COLORS=auto 2024-03-26T04:19:48.757 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.757 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:19:48.757 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.757 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:19:48.757 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.757 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.757 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/locale 2024-03-26T04:19:48.757 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:19:48.757 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.757 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:19:48.757 INFO:tasks.workunit.client.0.smithi016.stderr:++ echo api_c_write_operations 2024-03-26T04:19:48.757 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:19:48.758 INFO:tasks.workunit.client.0.smithi016.stderr:++ awk '{print $1}' 2024-03-26T04:19:48.758 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:19:48.758 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.758 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:19:48.758 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.758 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:19:48.758 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:19:48.758 INFO:tasks.workunit.client.0.smithi016.stderr:+++++ readlink /proc/74624/exe 2024-03-26T04:19:48.758 INFO:tasks.workunit.client.0.smithi016.stderr:+ ff=api_c_write_operations 2024-03-26T04:19:48.758 INFO:tasks.workunit.client.0.smithi016.stdout:test api_c_write_operations on pid 74823 2024-03-26T04:19:48.761 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74823 2024-03-26T04:19:48.761 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'test api_c_write_operations on pid 74823' 2024-03-26T04:19:48.761 INFO:tasks.workunit.client.0.smithi016.stderr:+ pids[$f]=74823 2024-03-26T04:19:48.761 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:19:48.761 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:19:48.761 INFO:tasks.workunit.client.0.smithi016.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-03-26T04:19:48.761 INFO:tasks.workunit.client.0.smithi016.stderr:++ printf %25s api_c_read_operations 2024-03-26T04:19:48.761 INFO:tasks.workunit.client.0.smithi016.stderr:+ r=' api_c_read_operations' 2024-03-26T04:19:48.761 INFO:tasks.workunit.client.0.smithi016.stderr:++++ basename /usr/bin/bash 2024-03-26T04:19:48.761 INFO:tasks.workunit.client.0.smithi016.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:19:48.762 INFO:tasks.workunit.client.0.smithi016.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:19:48.762 INFO:tasks.workunit.client.0.smithi016.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:19:48.762 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.762 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:19:48.762 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.762 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:19:48.762 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup config 2024-03-26T04:19:48.762 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.762 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.762 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:19:48.762 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.762 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:19:48.762 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.762 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:19:48.763 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:19:48.763 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.763 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:19:48.763 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.763 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:19:48.763 INFO:tasks.workunit.client.0.smithi016.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:19:48.763 INFO:tasks.workunit.client.0.smithi016.stderr:++ awk '{print $1}' 2024-03-26T04:19:48.763 INFO:tasks.workunit.client.0.smithi016.stderr:++ echo api_c_read_operations 2024-03-26T04:19:48.763 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:19:48.763 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_declare='declare -f' 2024-03-26T04:19:48.763 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_opt=-f 2024-03-26T04:19:48.763 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export which_declare 2024-03-26T04:19:48.763 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export -f which 2024-03-26T04:19:48.763 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:19:48.763 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset i 2024-03-26T04:19:48.764 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset -f pathmunge 2024-03-26T04:19:48.764 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:19:48.764 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:19:48.764 INFO:tasks.workunit.client.0.smithi016.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:19:48.764 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.764 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:19:48.764 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.764 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:19:48.764 INFO:tasks.workunit.client.0.smithi016.stderr:+ export PATH 2024-03-26T04:19:48.764 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:19:48.764 INFO:tasks.workunit.client.0.smithi016.stderr:+ unset rc 2024-03-26T04:19:48.764 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:19:48.764 INFO:tasks.workunit.client.0.smithi016.stderr:+ . /etc/bashrc 2024-03-26T04:19:48.764 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:19:48.764 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -z '' ']' 2024-03-26T04:19:48.765 INFO:tasks.workunit.client.0.smithi016.stderr:+++ return 2024-03-26T04:19:48.765 INFO:tasks.workunit.client.0.smithi016.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:19:48.765 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.765 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.765 INFO:tasks.workunit.client.0.smithi016.stderr:++ shopt -q login_shell 2024-03-26T04:19:48.765 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:19:48.765 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.765 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:19:48.765 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export S_COLORS=auto 2024-03-26T04:19:48.765 INFO:tasks.workunit.client.0.smithi016.stderr:+++ S_COLORS=auto 2024-03-26T04:19:48.765 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.765 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:19:48.765 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.765 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:19:48.765 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.766 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:19:48.766 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:19:48.766 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:19:48.766 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.766 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:19:48.766 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.766 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:19:48.766 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:19:48.766 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.766 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2024-03-26T04:19:48.766 INFO:tasks.workunit.client.0.smithi016.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:19:48.766 INFO:tasks.workunit.client.0.smithi016.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:19:48.766 INFO:tasks.workunit.client.0.smithi016.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:19:48.766 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.767 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:19:48.767 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.767 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:19:48.767 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup config 2024-03-26T04:19:48.767 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.767 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.767 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:19:48.767 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.767 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:19:48.767 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.767 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:19:48.767 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:19:48.767 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.767 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:19:48.767 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.768 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:19:48.768 INFO:tasks.workunit.client.0.smithi016.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:19:48.768 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:19:48.768 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:19:48.768 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:19:48.768 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.768 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:19:48.768 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.768 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:19:48.768 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:19:48.768 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.768 INFO:tasks.workunit.client.0.smithi016.stderr:+ sed 's/^/ api_watch_notify: /' 2024-03-26T04:19:48.769 INFO:tasks.workunit.client.0.smithi016.stdout:test api_c_read_operations on pid 74863 2024-03-26T04:19:48.774 INFO:tasks.workunit.client.0.smithi016.stderr:+ ff=api_c_read_operations 2024-03-26T04:19:48.774 INFO:tasks.workunit.client.0.smithi016.stderr:+++++ readlink /proc/74638/exe 2024-03-26T04:19:48.774 INFO:tasks.workunit.client.0.smithi016.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-03-26T04:19:48.774 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74863 2024-03-26T04:19:48.774 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'test api_c_read_operations on pid 74863' 2024-03-26T04:19:48.774 INFO:tasks.workunit.client.0.smithi016.stderr:+ pids[$f]=74863 2024-03-26T04:19:48.774 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:19:48.774 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:19:48.774 INFO:tasks.workunit.client.0.smithi016.stderr:+++ umask 2024-03-26T04:19:48.775 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:19:48.775 INFO:tasks.workunit.client.0.smithi016.stderr:++++ basename /usr/bin/bash 2024-03-26T04:19:48.775 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:19:48.775 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:19:48.775 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.775 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:19:48.775 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.775 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:19:48.775 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:19:48.775 INFO:tasks.workunit.client.0.smithi016.stderr:++ SHELL=/bin/bash 2024-03-26T04:19:48.775 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:19:48.775 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.775 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:19:48.775 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.775 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:19:48.776 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.776 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:19:48.776 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.776 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:19:48.776 INFO:tasks.workunit.client.0.smithi016.stderr:++ printf %25s api_cls_remote_reads 2024-03-26T04:19:48.776 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.776 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:19:48.776 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.776 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:19:48.776 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup 2024-03-26T04:19:48.776 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:19:48.776 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.776 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:19:48.776 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.777 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:19:48.777 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.777 INFO:tasks.workunit.client.0.smithi016.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.777 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.777 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:19:48.777 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.777 INFO:tasks.workunit.client.0.smithi016.stderr:+ r=' api_cls_remote_reads' 2024-03-26T04:19:48.777 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:19:48.777 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.777 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/locale 2024-03-26T04:19:48.777 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_declare='declare -f' 2024-03-26T04:19:48.777 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_opt=-f 2024-03-26T04:19:48.777 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export which_declare 2024-03-26T04:19:48.777 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export -f which 2024-03-26T04:19:48.778 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset i 2024-03-26T04:19:48.778 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset -f pathmunge 2024-03-26T04:19:48.778 INFO:tasks.workunit.client.0.smithi016.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:19:48.778 INFO:tasks.workunit.client.0.smithi016.stderr:+ export PATH 2024-03-26T04:19:48.778 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:19:48.778 INFO:tasks.workunit.client.0.smithi016.stderr:+ unset rc 2024-03-26T04:19:48.778 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2024-03-26T04:19:48.778 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.778 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:19:48.778 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.778 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:19:48.778 INFO:tasks.workunit.client.0.smithi016.stderr:+ sed 's/^/ api_cmd: /' 2024-03-26T04:19:48.778 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:19:48.778 INFO:tasks.workunit.client.0.smithi016.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-03-26T04:19:48.779 INFO:tasks.workunit.client.0.smithi016.stderr:+ tee ceph_test_rados_api_cmd.log 2024-03-26T04:19:48.779 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:19:48.779 INFO:tasks.workunit.client.0.smithi016.stderr:+ . /etc/bashrc 2024-03-26T04:19:48.779 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:19:48.779 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:19:48.779 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:19:48.779 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -z '' ']' 2024-03-26T04:19:48.779 INFO:tasks.workunit.client.0.smithi016.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:19:48.779 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.779 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.779 INFO:tasks.workunit.client.0.smithi016.stderr:++ shopt -q login_shell 2024-03-26T04:19:48.779 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:19:48.779 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.779 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:19:48.780 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:19:48.780 INFO:tasks.workunit.client.0.smithi016.stderr:+++ return 2024-03-26T04:19:48.780 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.780 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:19:48.780 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.780 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:19:48.780 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export S_COLORS=auto 2024-03-26T04:19:48.780 INFO:tasks.workunit.client.0.smithi016.stderr:+++ S_COLORS=auto 2024-03-26T04:19:48.780 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.780 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:19:48.780 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.780 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:19:48.780 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.780 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:19:48.780 INFO:tasks.workunit.client.0.smithi016.stderr:++ awk '{print $1}' 2024-03-26T04:19:48.781 INFO:tasks.workunit.client.0.smithi016.stderr:+++++ readlink /proc/74709/exe 2024-03-26T04:19:48.781 INFO:tasks.workunit.client.0.smithi016.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:19:48.781 INFO:tasks.workunit.client.0.smithi016.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:19:48.781 INFO:tasks.workunit.client.0.smithi016.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:19:48.781 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.781 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:19:48.781 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.781 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:19:48.781 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup config 2024-03-26T04:19:48.781 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.781 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.781 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:19:48.781 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.781 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:19:48.782 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.782 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:19:48.782 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:19:48.782 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.782 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:19:48.782 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.782 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:19:48.782 INFO:tasks.workunit.client.0.smithi016.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:19:48.782 INFO:tasks.workunit.client.0.smithi016.stderr:+++ umask 2024-03-26T04:19:48.782 INFO:tasks.workunit.client.0.smithi016.stderr:++++ basename /usr/bin/bash 2024-03-26T04:19:48.782 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:19:48.782 INFO:tasks.workunit.client.0.smithi016.stderr:++ SHELL=/bin/bash 2024-03-26T04:19:48.782 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.782 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:19:48.783 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.783 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:19:48.783 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:19:48.783 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.783 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:19:48.783 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.783 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:19:48.783 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_declare='declare -f' 2024-03-26T04:19:48.783 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.783 INFO:tasks.workunit.client.0.smithi016.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.783 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_opt=-f 2024-03-26T04:19:48.783 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.783 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:19:48.783 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.783 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:19:48.784 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export which_declare 2024-03-26T04:19:48.784 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export -f which 2024-03-26T04:19:48.784 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.784 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset i 2024-03-26T04:19:48.784 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset -f pathmunge 2024-03-26T04:19:48.784 INFO:tasks.workunit.client.0.smithi016.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:19:48.784 INFO:tasks.workunit.client.0.smithi016.stderr:+ export PATH 2024-03-26T04:19:48.784 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:19:48.784 INFO:tasks.workunit.client.0.smithi016.stderr:+ unset rc 2024-03-26T04:19:48.784 INFO:tasks.workunit.client.0.smithi016.stderr:++ echo api_cls_remote_reads 2024-03-26T04:19:48.784 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2024-03-26T04:19:48.784 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:19:48.784 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:19:48.784 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:19:48.785 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.785 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:19:48.785 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.785 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:19:48.785 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:19:48.785 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.785 INFO:tasks.workunit.client.0.smithi016.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-03-26T04:19:48.785 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:19:48.785 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:19:48.785 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:19:48.785 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.785 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:19:48.785 INFO:tasks.workunit.client.0.smithi016.stderr:+++++ readlink /proc/74743/exe 2024-03-26T04:19:48.785 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.785 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:19:48.786 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:19:48.786 INFO:tasks.workunit.client.0.smithi016.stderr:+++ return 2024-03-26T04:19:48.786 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.786 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:19:48.786 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.786 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:19:48.786 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export S_COLORS=auto 2024-03-26T04:19:48.786 INFO:tasks.workunit.client.0.smithi016.stderr:+++ S_COLORS=auto 2024-03-26T04:19:48.786 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.786 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:19:48.786 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.786 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:19:48.786 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.787 INFO:tasks.workunit.client.0.smithi016.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-03-26T04:19:48.787 INFO:tasks.workunit.client.0.smithi016.stdout:test api_cls_remote_reads on pid 74896 2024-03-26T04:19:48.788 INFO:tasks.workunit.client.0.smithi016.stderr:++++ basename /usr/bin/bash 2024-03-26T04:19:48.788 INFO:tasks.workunit.client.0.smithi016.stderr:+ ff=api_cls_remote_reads 2024-03-26T04:19:48.788 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74896 2024-03-26T04:19:48.788 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'test api_cls_remote_reads on pid 74896' 2024-03-26T04:19:48.788 INFO:tasks.workunit.client.0.smithi016.stderr:+ pids[$f]=74896 2024-03-26T04:19:48.789 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:19:48.789 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:19:48.789 INFO:tasks.workunit.client.0.smithi016.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2024-03-26T04:19:48.789 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:19:48.789 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:19:48.789 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:19:48.789 INFO:tasks.workunit.client.0.smithi016.stderr:++ printf %25s list_parallel 2024-03-26T04:19:48.789 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.789 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:19:48.789 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_declare='declare -f' 2024-03-26T04:19:48.789 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.789 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_opt=-f 2024-03-26T04:19:48.789 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:19:48.789 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export which_declare 2024-03-26T04:19:48.790 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export -f which 2024-03-26T04:19:48.790 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:19:48.790 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.790 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset i 2024-03-26T04:19:48.790 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:19:48.790 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset -f pathmunge 2024-03-26T04:19:48.790 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.790 INFO:tasks.workunit.client.0.smithi016.stderr:+ r=' list_parallel' 2024-03-26T04:19:48.790 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:19:48.790 INFO:tasks.workunit.client.0.smithi016.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:19:48.790 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.790 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:19:48.790 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.791 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:19:48.791 INFO:tasks.workunit.client.0.smithi016.stderr:+ export PATH 2024-03-26T04:19:48.791 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup 2024-03-26T04:19:48.791 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:19:48.791 INFO:tasks.workunit.client.0.smithi016.stderr:+ unset rc 2024-03-26T04:19:48.791 INFO:tasks.workunit.client.0.smithi016.stderr:++ echo list_parallel 2024-03-26T04:19:48.791 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/locale 2024-03-26T04:19:48.791 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:19:48.791 INFO:tasks.workunit.client.0.smithi016.stderr:+ tee ceph_test_rados_api_service.log 2024-03-26T04:19:48.791 INFO:tasks.workunit.client.0.smithi016.stderr:+ sed 's/^/ api_service: /' 2024-03-26T04:19:48.791 INFO:tasks.workunit.client.0.smithi016.stderr:++ awk '{print $1}' 2024-03-26T04:19:48.791 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:19:48.791 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:19:48.791 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:19:48.792 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.792 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:19:48.792 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.792 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:19:48.792 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:19:48.792 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.792 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:19:48.792 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.792 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:19:48.792 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.792 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:19:48.792 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.792 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:19:48.792 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup 2024-03-26T04:19:48.793 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.793 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:19:48.793 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.793 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:19:48.793 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:19:48.793 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:19:48.793 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:19:48.793 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:19:48.793 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.793 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:19:48.793 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.793 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:19:48.793 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:19:48.794 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.794 INFO:tasks.workunit.client.0.smithi016.stdout:test list_parallel on pid 74920 2024-03-26T04:19:48.801 INFO:tasks.workunit.client.0.smithi016.stderr:+ ff=list_parallel 2024-03-26T04:19:48.801 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74920 2024-03-26T04:19:48.801 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'test list_parallel on pid 74920' 2024-03-26T04:19:48.802 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:19:48.802 INFO:tasks.workunit.client.0.smithi016.stderr:+ pids[$f]=74920 2024-03-26T04:19:48.802 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:19:48.802 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:19:48.802 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2024-03-26T04:19:48.802 INFO:tasks.workunit.client.0.smithi016.stderr:+ . /etc/bashrc 2024-03-26T04:19:48.802 INFO:tasks.workunit.client.0.smithi016.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-03-26T04:19:48.802 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -z '' ']' 2024-03-26T04:19:48.802 INFO:tasks.workunit.client.0.smithi016.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:19:48.802 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.802 INFO:tasks.workunit.client.0.smithi016.stderr:++ shopt -q login_shell 2024-03-26T04:19:48.802 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/locale 2024-03-26T04:19:48.803 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:19:48.803 INFO:tasks.workunit.client.0.smithi016.stderr:+ . /etc/bashrc 2024-03-26T04:19:48.803 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -z '' ']' 2024-03-26T04:19:48.803 INFO:tasks.workunit.client.0.smithi016.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:19:48.803 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.803 INFO:tasks.workunit.client.0.smithi016.stderr:++ shopt -q login_shell 2024-03-26T04:19:48.803 INFO:tasks.workunit.client.0.smithi016.stderr:+++ umask 2024-03-26T04:19:48.803 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:19:48.803 INFO:tasks.workunit.client.0.smithi016.stderr:++ SHELL=/bin/bash 2024-03-26T04:19:48.803 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.803 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:19:48.803 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.803 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:19:48.803 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:19:48.804 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.804 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:19:48.804 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.804 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:19:48.804 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.804 INFO:tasks.workunit.client.0.smithi016.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.804 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.804 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:19:48.804 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.804 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:19:48.804 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.804 INFO:tasks.workunit.client.0.smithi016.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:19:48.804 INFO:tasks.workunit.client.0.smithi016.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:19:48.805 INFO:tasks.workunit.client.0.smithi016.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:19:48.805 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.805 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:19:48.805 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.805 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:19:48.805 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup config 2024-03-26T04:19:48.805 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.805 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.805 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:19:48.805 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.805 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:19:48.805 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.805 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:19:48.805 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:19:48.806 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.806 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:19:48.806 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.806 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:19:48.806 INFO:tasks.workunit.client.0.smithi016.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:19:48.806 INFO:tasks.workunit.client.0.smithi016.stderr:++ printf %25s open_pools_parallel 2024-03-26T04:19:48.806 INFO:tasks.workunit.client.0.smithi016.stderr:+ r=' open_pools_parallel' 2024-03-26T04:19:48.806 INFO:tasks.workunit.client.0.smithi016.stderr:+++++ readlink /proc/74770/exe 2024-03-26T04:19:48.806 INFO:tasks.workunit.client.0.smithi016.stderr:++++ basename /usr/bin/bash 2024-03-26T04:19:48.806 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:19:48.806 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:19:48.806 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:19:48.806 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.806 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:19:48.807 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.807 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:19:48.807 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:19:48.807 INFO:tasks.workunit.client.0.smithi016.stderr:++ echo open_pools_parallel 2024-03-26T04:19:48.807 INFO:tasks.workunit.client.0.smithi016.stderr:+++ return 2024-03-26T04:19:48.807 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.807 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:19:48.807 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.807 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:19:48.807 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export S_COLORS=auto 2024-03-26T04:19:48.807 INFO:tasks.workunit.client.0.smithi016.stderr:+++ S_COLORS=auto 2024-03-26T04:19:48.807 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.807 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:19:48.807 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.808 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:19:48.808 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.808 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_declare='declare -f' 2024-03-26T04:19:48.808 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_opt=-f 2024-03-26T04:19:48.808 INFO:tasks.workunit.client.0.smithi016.stderr:+++ umask 2024-03-26T04:19:48.808 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export which_declare 2024-03-26T04:19:48.808 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export -f which 2024-03-26T04:19:48.808 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset i 2024-03-26T04:19:48.808 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset -f pathmunge 2024-03-26T04:19:48.808 INFO:tasks.workunit.client.0.smithi016.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:19:48.808 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:19:48.808 INFO:tasks.workunit.client.0.smithi016.stderr:+ export PATH 2024-03-26T04:19:48.808 INFO:tasks.workunit.client.0.smithi016.stderr:++ SHELL=/bin/bash 2024-03-26T04:19:48.808 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:19:48.809 INFO:tasks.workunit.client.0.smithi016.stderr:+ unset rc 2024-03-26T04:19:48.809 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.809 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:19:48.809 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.809 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:19:48.809 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:19:48.809 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.809 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:19:48.809 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.809 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:19:48.809 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.809 INFO:tasks.workunit.client.0.smithi016.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.809 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.809 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:19:48.810 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.810 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:19:48.810 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.810 INFO:tasks.workunit.client.0.smithi016.stderr:+ sed 's/^/ api_service_pp: /' 2024-03-26T04:19:48.810 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:19:48.810 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:19:48.810 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:19:48.810 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.810 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:19:48.810 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.810 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:19:48.810 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:19:48.810 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.810 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:19:48.811 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.811 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:19:48.811 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.811 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:19:48.811 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.811 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:19:48.811 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup 2024-03-26T04:19:48.811 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:19:48.811 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:19:48.811 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:19:48.811 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.811 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:19:48.811 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.811 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:19:48.812 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:19:48.812 INFO:tasks.workunit.client.0.smithi016.stderr:+++ return 2024-03-26T04:19:48.812 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.812 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:19:48.812 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.812 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:19:48.812 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export S_COLORS=auto 2024-03-26T04:19:48.812 INFO:tasks.workunit.client.0.smithi016.stderr:+++ S_COLORS=auto 2024-03-26T04:19:48.812 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.812 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:19:48.812 INFO:tasks.workunit.client.0.smithi016.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-03-26T04:19:48.812 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.812 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:19:48.812 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.813 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/locale 2024-03-26T04:19:48.813 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.813 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:19:48.813 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.813 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:19:48.813 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:19:48.813 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:19:48.813 INFO:tasks.workunit.client.0.smithi016.stderr:++ awk '{print $1}' 2024-03-26T04:19:48.813 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2024-03-26T04:19:48.813 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.813 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:19:48.813 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.813 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:19:48.813 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:19:48.814 INFO:tasks.workunit.client.0.smithi016.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:19:48.814 INFO:tasks.workunit.client.0.smithi016.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:19:48.814 INFO:tasks.workunit.client.0.smithi016.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:19:48.814 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.814 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:19:48.814 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.814 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:19:48.814 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup config 2024-03-26T04:19:48.814 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.814 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:19:48.814 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:19:48.814 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.814 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:19:48.814 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:19:48.815 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.815 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.815 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:19:48.815 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:19:48.815 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.815 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:19:48.815 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.815 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:19:48.815 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:19:48.815 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:19:48.815 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.815 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.815 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:19:48.815 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.815 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:19:48.816 INFO:tasks.workunit.client.0.smithi016.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:19:48.816 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:19:48.816 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:19:48.816 INFO:tasks.workunit.client.0.smithi016.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:19:48.816 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:19:48.816 INFO:tasks.workunit.client.0.smithi016.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:19:48.816 INFO:tasks.workunit.client.0.smithi016.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:19:48.816 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:19:48.816 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.816 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.816 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:19:48.816 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:19:48.816 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.816 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:19:48.816 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.817 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:19:48.817 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:19:48.817 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup config 2024-03-26T04:19:48.817 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.817 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.817 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.817 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:19:48.817 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.817 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:19:48.817 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.817 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:19:48.817 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:19:48.817 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.817 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:19:48.817 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.818 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:19:48.818 INFO:tasks.workunit.client.0.smithi016.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:19:48.818 INFO:tasks.workunit.client.0.smithi016.stderr:+++++ readlink /proc/74823/exe 2024-03-26T04:19:48.818 INFO:tasks.workunit.client.0.smithi016.stderr:+++++ readlink /proc/74863/exe 2024-03-26T04:19:48.818 INFO:tasks.workunit.client.0.smithi016.stderr:++++ basename /usr/bin/bash 2024-03-26T04:19:48.818 INFO:tasks.workunit.client.0.smithi016.stdout:test open_pools_parallel on pid 74980 2024-03-26T04:19:48.821 INFO:tasks.workunit.client.0.smithi016.stderr:+ ff=open_pools_parallel 2024-03-26T04:19:48.821 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74980 2024-03-26T04:19:48.821 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'test open_pools_parallel on pid 74980' 2024-03-26T04:19:48.821 INFO:tasks.workunit.client.0.smithi016.stderr:+ pids[$f]=74980 2024-03-26T04:19:48.821 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-03-26T04:19:48.821 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:19:48.821 INFO:tasks.workunit.client.0.smithi016.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-03-26T04:19:48.822 INFO:tasks.workunit.client.0.smithi016.stderr:++ printf %25s delete_pools_parallel 2024-03-26T04:19:48.822 INFO:tasks.workunit.client.0.smithi016.stderr:++++ basename /usr/bin/bash 2024-03-26T04:19:48.822 INFO:tasks.workunit.client.0.smithi016.stderr:+ r=' delete_pools_parallel' 2024-03-26T04:19:48.822 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:19:48.822 INFO:tasks.workunit.client.0.smithi016.stderr:+ . /etc/bashrc 2024-03-26T04:19:48.822 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -z '' ']' 2024-03-26T04:19:48.822 INFO:tasks.workunit.client.0.smithi016.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:19:48.822 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.822 INFO:tasks.workunit.client.0.smithi016.stderr:++ shopt -q login_shell 2024-03-26T04:19:48.822 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_declare='declare -f' 2024-03-26T04:19:48.822 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_opt=-f 2024-03-26T04:19:48.822 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export which_declare 2024-03-26T04:19:48.822 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export -f which 2024-03-26T04:19:48.823 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset i 2024-03-26T04:19:48.823 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset -f pathmunge 2024-03-26T04:19:48.823 INFO:tasks.workunit.client.0.smithi016.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:19:48.823 INFO:tasks.workunit.client.0.smithi016.stderr:+++ umask 2024-03-26T04:19:48.823 INFO:tasks.workunit.client.0.smithi016.stderr:+ export PATH 2024-03-26T04:19:48.823 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:19:48.823 INFO:tasks.workunit.client.0.smithi016.stderr:+ unset rc 2024-03-26T04:19:48.823 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:19:48.823 INFO:tasks.workunit.client.0.smithi016.stderr:++ SHELL=/bin/bash 2024-03-26T04:19:48.823 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.823 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:19:48.823 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.823 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:19:48.823 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:19:48.824 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.824 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:19:48.824 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.824 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:19:48.824 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_declare='declare -f' 2024-03-26T04:19:48.824 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.824 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_opt=-f 2024-03-26T04:19:48.824 INFO:tasks.workunit.client.0.smithi016.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.824 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2024-03-26T04:19:48.824 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.824 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:19:48.824 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.824 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export which_declare 2024-03-26T04:19:48.824 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:19:48.825 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export -f which 2024-03-26T04:19:48.825 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.825 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset i 2024-03-26T04:19:48.825 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset -f pathmunge 2024-03-26T04:19:48.825 INFO:tasks.workunit.client.0.smithi016.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:19:48.825 INFO:tasks.workunit.client.0.smithi016.stderr:+ export PATH 2024-03-26T04:19:48.825 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:19:48.825 INFO:tasks.workunit.client.0.smithi016.stderr:+ unset rc 2024-03-26T04:19:48.825 INFO:tasks.workunit.client.0.smithi016.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-03-26T04:19:48.825 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:19:48.825 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:19:48.825 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:19:48.825 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.825 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:19:48.826 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.826 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:19:48.826 INFO:tasks.workunit.client.0.smithi016.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-03-26T04:19:48.826 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:19:48.826 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.826 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:19:48.826 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.826 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:19:48.826 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.826 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:19:48.826 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.826 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:19:48.826 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup 2024-03-26T04:19:48.826 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:19:48.827 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:19:48.827 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:19:48.827 INFO:tasks.workunit.client.0.smithi016.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-03-26T04:19:48.827 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.827 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:19:48.827 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.827 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:19:48.827 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:19:48.827 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.827 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:19:48.827 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.827 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:19:48.827 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/locale 2024-03-26T04:19:48.827 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.828 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:19:48.828 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.828 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:19:48.828 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup 2024-03-26T04:19:48.828 INFO:tasks.workunit.client.0.smithi016.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-03-26T04:19:48.828 INFO:tasks.workunit.client.0.smithi016.stderr:++ echo delete_pools_parallel 2024-03-26T04:19:48.828 INFO:tasks.workunit.client.0.smithi016.stderr:++ awk '{print $1}' 2024-03-26T04:19:48.828 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2024-03-26T04:19:48.828 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.828 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:19:48.828 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.828 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:19:48.828 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:19:48.828 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/locale 2024-03-26T04:19:48.829 INFO:tasks.workunit.client.0.smithi016.stdout:test delete_pools_parallel on pid 75016 2024-03-26T04:19:48.835 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.835 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:19:48.835 INFO:tasks.workunit.client.0.smithi016.stderr:+ ff=delete_pools_parallel 2024-03-26T04:19:48.835 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.835 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:19:48.835 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:19:48.835 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=75016 2024-03-26T04:19:48.835 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'test delete_pools_parallel on pid 75016' 2024-03-26T04:19:48.835 INFO:tasks.workunit.client.0.smithi016.stderr:+ pids[$f]=75016 2024-03-26T04:19:48.835 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T04:19:48.835 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:19:48.835 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:19:48.835 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:19:48.836 INFO:tasks.workunit.client.0.smithi016.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-03-26T04:19:48.836 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:19:48.836 INFO:tasks.workunit.client.0.smithi016.stderr:+ . /etc/bashrc 2024-03-26T04:19:48.836 INFO:tasks.workunit.client.0.smithi016.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:19:48.836 INFO:tasks.workunit.client.0.smithi016.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:19:48.836 INFO:tasks.workunit.client.0.smithi016.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:19:48.836 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.836 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:19:48.836 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -z '' ']' 2024-03-26T04:19:48.836 INFO:tasks.workunit.client.0.smithi016.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:19:48.836 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.836 INFO:tasks.workunit.client.0.smithi016.stderr:++ shopt -q login_shell 2024-03-26T04:19:48.836 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.836 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:19:48.837 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup config 2024-03-26T04:19:48.837 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.837 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.837 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:19:48.837 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.837 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:19:48.837 INFO:tasks.workunit.client.0.smithi016.stderr:+++ umask 2024-03-26T04:19:48.837 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.837 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:19:48.837 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:19:48.837 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.837 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:19:48.837 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.837 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:19:48.838 INFO:tasks.workunit.client.0.smithi016.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:19:48.838 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:19:48.838 INFO:tasks.workunit.client.0.smithi016.stderr:++ SHELL=/bin/bash 2024-03-26T04:19:48.838 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.838 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:19:48.838 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.838 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:19:48.838 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:19:48.838 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.838 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:19:48.838 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.838 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:19:48.838 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.838 INFO:tasks.workunit.client.0.smithi016.stderr:+++++ readlink /proc/74896/exe 2024-03-26T04:19:48.839 INFO:tasks.workunit.client.0.smithi016.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.839 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.839 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:19:48.839 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.839 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:19:48.839 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.839 INFO:tasks.workunit.client.0.smithi016.stderr:++++ basename /usr/bin/bash 2024-03-26T04:19:48.839 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:19:48.839 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:19:48.839 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:19:48.839 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.839 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:19:48.839 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.840 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:19:48.840 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:19:48.840 INFO:tasks.workunit.client.0.smithi016.stderr:+++ return 2024-03-26T04:19:48.840 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.840 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:19:48.840 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.840 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:19:48.840 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export S_COLORS=auto 2024-03-26T04:19:48.840 INFO:tasks.workunit.client.0.smithi016.stderr:+++ S_COLORS=auto 2024-03-26T04:19:48.840 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.840 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:19:48.840 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.840 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:19:48.840 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.841 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_declare='declare -f' 2024-03-26T04:19:48.841 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_opt=-f 2024-03-26T04:19:48.841 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export which_declare 2024-03-26T04:19:48.841 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export -f which 2024-03-26T04:19:48.841 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset i 2024-03-26T04:19:48.841 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset -f pathmunge 2024-03-26T04:19:48.841 INFO:tasks.workunit.client.0.smithi016.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:19:48.841 INFO:tasks.workunit.client.0.smithi016.stderr:+ export PATH 2024-03-26T04:19:48.841 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:19:48.841 INFO:tasks.workunit.client.0.smithi016.stderr:+ unset rc 2024-03-26T04:19:48.841 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph_test_rados_api_cls_remote_reads --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cls_remote_reads.xml 2024-03-26T04:19:48.841 INFO:tasks.workunit.client.0.smithi016.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:19:48.841 INFO:tasks.workunit.client.0.smithi016.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:19:48.841 INFO:tasks.workunit.client.0.smithi016.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:19:48.841 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.842 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:19:48.842 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.842 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:19:48.842 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup config 2024-03-26T04:19:48.842 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.842 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.842 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:19:48.842 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.842 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:19:48.842 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.842 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:19:48.842 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:19:48.842 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.842 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:19:48.843 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.843 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:19:48.843 INFO:tasks.workunit.client.0.smithi016.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:19:48.843 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:19:48.843 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:19:48.843 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:19:48.843 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.843 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:19:48.843 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.843 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:19:48.843 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:19:48.843 INFO:tasks.workunit.client.0.smithi016.stderr:+++ return 2024-03-26T04:19:48.843 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.843 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:19:48.844 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.844 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:19:48.844 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export S_COLORS=auto 2024-03-26T04:19:48.844 INFO:tasks.workunit.client.0.smithi016.stderr:++ printf %25s cls 2024-03-26T04:19:48.844 INFO:tasks.workunit.client.0.smithi016.stderr:+++ S_COLORS=auto 2024-03-26T04:19:48.844 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.844 INFO:tasks.workunit.client.0.smithi016.stderr:+++++ readlink /proc/74920/exe 2024-03-26T04:19:48.844 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:19:48.844 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.844 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:19:48.844 INFO:tasks.workunit.client.0.smithi016.stderr:+ r=' cls' 2024-03-26T04:19:48.844 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.844 INFO:tasks.workunit.client.0.smithi016.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2024-03-26T04:19:48.844 INFO:tasks.workunit.client.0.smithi016.stderr:++++ basename /usr/bin/bash 2024-03-26T04:19:48.844 INFO:tasks.workunit.client.0.smithi016.stderr:++ echo cls 2024-03-26T04:19:48.845 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_declare='declare -f' 2024-03-26T04:19:48.845 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_opt=-f 2024-03-26T04:19:48.845 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export which_declare 2024-03-26T04:19:48.845 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export -f which 2024-03-26T04:19:48.845 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset i 2024-03-26T04:19:48.845 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset -f pathmunge 2024-03-26T04:19:48.845 INFO:tasks.workunit.client.0.smithi016.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:19:48.845 INFO:tasks.workunit.client.0.smithi016.stderr:+ export PATH 2024-03-26T04:19:48.845 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:19:48.845 INFO:tasks.workunit.client.0.smithi016.stderr:+ unset rc 2024-03-26T04:19:48.845 INFO:tasks.workunit.client.0.smithi016.stderr:+ tee ceph_test_rados_list_parallel.log 2024-03-26T04:19:48.845 INFO:tasks.workunit.client.0.smithi016.stderr:+ sed 's/^/ list_parallel: /' 2024-03-26T04:19:48.845 INFO:tasks.workunit.client.0.smithi016.stderr:++ awk '{print $1}' 2024-03-26T04:19:48.845 INFO:tasks.workunit.client.0.smithi016.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2024-03-26T04:19:48.846 INFO:tasks.workunit.client.0.smithi016.stdout:test cls on pid 75055 2024-03-26T04:19:48.848 INFO:tasks.workunit.client.0.smithi016.stderr:+ ff=cls 2024-03-26T04:19:48.848 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=75055 2024-03-26T04:19:48.848 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'test cls on pid 75055' 2024-03-26T04:19:48.848 INFO:tasks.workunit.client.0.smithi016.stderr:+ pids[$f]=75055 2024-03-26T04:19:48.848 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T04:19:48.848 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:19:48.848 INFO:tasks.workunit.client.0.smithi016.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2024-03-26T04:19:48.848 INFO:tasks.workunit.client.0.smithi016.stderr:++ printf %25s cmd 2024-03-26T04:19:48.848 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:19:48.848 INFO:tasks.workunit.client.0.smithi016.stderr:+ r=' cmd' 2024-03-26T04:19:48.849 INFO:tasks.workunit.client.0.smithi016.stderr:+ . /etc/bashrc 2024-03-26T04:19:48.849 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -z '' ']' 2024-03-26T04:19:48.849 INFO:tasks.workunit.client.0.smithi016.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:19:48.849 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.849 INFO:tasks.workunit.client.0.smithi016.stderr:++ shopt -q login_shell 2024-03-26T04:19:48.849 INFO:tasks.workunit.client.0.smithi016.stderr:++ echo cmd 2024-03-26T04:19:48.849 INFO:tasks.workunit.client.0.smithi016.stderr:+++ umask 2024-03-26T04:19:48.849 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:19:48.849 INFO:tasks.workunit.client.0.smithi016.stderr:++ SHELL=/bin/bash 2024-03-26T04:19:48.849 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.849 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:19:48.849 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.849 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:19:48.850 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:19:48.850 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.850 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:19:48.850 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.850 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:19:48.850 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.850 INFO:tasks.workunit.client.0.smithi016.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.850 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.850 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:19:48.850 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.850 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:19:48.850 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.850 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2024-03-26T04:19:48.850 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:19:48.850 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:19:48.851 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:19:48.851 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.851 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:19:48.851 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.851 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:19:48.851 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:19:48.851 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.851 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:19:48.851 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:19:48.851 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:19:48.851 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.851 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:19:48.851 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.851 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:19:48.852 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:19:48.852 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.852 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:19:48.852 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.852 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:19:48.852 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.852 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:19:48.852 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.852 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:19:48.852 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup 2024-03-26T04:19:48.852 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/locale 2024-03-26T04:19:48.852 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:19:48.852 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:19:48.852 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:19:48.853 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.853 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:19:48.853 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.853 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:19:48.853 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:19:48.853 INFO:tasks.workunit.client.0.smithi016.stderr:+++ return 2024-03-26T04:19:48.853 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.853 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:19:48.853 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.853 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:19:48.853 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export S_COLORS=auto 2024-03-26T04:19:48.853 INFO:tasks.workunit.client.0.smithi016.stderr:+++ S_COLORS=auto 2024-03-26T04:19:48.853 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.853 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:19:48.853 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.854 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:19:48.854 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.854 INFO:tasks.workunit.client.0.smithi016.stderr:++ awk '{print $1}' 2024-03-26T04:19:48.854 INFO:tasks.workunit.client.0.smithi016.stdout:test cmd on pid 75075 2024-03-26T04:19:48.862 INFO:tasks.workunit.client.0.smithi016.stderr:+ ff=cmd 2024-03-26T04:19:48.862 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=75075 2024-03-26T04:19:48.862 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'test cmd on pid 75075' 2024-03-26T04:19:48.862 INFO:tasks.workunit.client.0.smithi016.stderr:+ pids[$f]=75075 2024-03-26T04:19:48.862 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T04:19:48.862 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:19:48.862 INFO:tasks.workunit.client.0.smithi016.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2024-03-26T04:19:48.862 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:19:48.863 INFO:tasks.workunit.client.0.smithi016.stderr:+ . /etc/bashrc 2024-03-26T04:19:48.863 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -z '' ']' 2024-03-26T04:19:48.863 INFO:tasks.workunit.client.0.smithi016.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:19:48.863 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.863 INFO:tasks.workunit.client.0.smithi016.stderr:++ shopt -q login_shell 2024-03-26T04:19:48.863 INFO:tasks.workunit.client.0.smithi016.stderr:+++ umask 2024-03-26T04:19:48.863 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:19:48.863 INFO:tasks.workunit.client.0.smithi016.stderr:++ SHELL=/bin/bash 2024-03-26T04:19:48.863 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.863 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:19:48.863 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.863 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:19:48.864 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:19:48.864 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.864 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:19:48.864 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.864 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:19:48.864 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.864 INFO:tasks.workunit.client.0.smithi016.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.864 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.864 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:19:48.864 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.864 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:19:48.864 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.864 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:19:48.864 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:19:48.864 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:19:48.865 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.865 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:19:48.865 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.865 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:19:48.865 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:19:48.865 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.865 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:19:48.865 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:19:48.865 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:19:48.865 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.865 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:19:48.865 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.865 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:19:48.865 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:19:48.866 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.866 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:19:48.866 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:19:48.866 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:19:48.866 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.866 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:19:48.866 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.866 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:19:48.866 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:19:48.866 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.866 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:19:48.866 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.866 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:19:48.866 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.867 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:19:48.867 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.867 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:19:48.867 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup 2024-03-26T04:19:48.867 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:19:48.867 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:19:48.867 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:19:48.867 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.867 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:19:48.867 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.867 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:19:48.867 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:19:48.867 INFO:tasks.workunit.client.0.smithi016.stderr:+++ return 2024-03-26T04:19:48.867 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.867 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:19:48.868 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.868 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:19:48.868 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export S_COLORS=auto 2024-03-26T04:19:48.868 INFO:tasks.workunit.client.0.smithi016.stderr:+++ S_COLORS=auto 2024-03-26T04:19:48.868 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.868 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:19:48.868 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.868 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:19:48.868 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.868 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/locale 2024-03-26T04:19:48.868 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.868 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:19:48.868 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.868 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:19:48.868 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:19:48.869 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:19:48.869 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:19:48.869 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:19:48.869 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:19:48.869 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.869 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:19:48.869 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.869 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:19:48.869 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:19:48.869 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.869 INFO:tasks.workunit.client.0.smithi016.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:19:48.869 INFO:tasks.workunit.client.0.smithi016.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:19:48.869 INFO:tasks.workunit.client.0.smithi016.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:19:48.869 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.869 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:19:48.870 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.870 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:19:48.870 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.870 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup config 2024-03-26T04:19:48.870 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:19:48.870 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.870 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.870 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:19:48.870 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:19:48.870 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.870 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:19:48.870 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.870 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:19:48.870 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.871 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:19:48.871 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:19:48.871 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.871 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:19:48.871 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.871 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:19:48.871 INFO:tasks.workunit.client.0.smithi016.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:19:48.871 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:19:48.871 INFO:tasks.workunit.client.0.smithi016.stderr:++ printf %25s handler_error 2024-03-26T04:19:48.871 INFO:tasks.workunit.client.0.smithi016.stderr:+ r=' handler_error' 2024-03-26T04:19:48.871 INFO:tasks.workunit.client.0.smithi016.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:19:48.871 INFO:tasks.workunit.client.0.smithi016.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:19:48.871 INFO:tasks.workunit.client.0.smithi016.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:19:48.871 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.871 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:19:48.872 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.872 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:19:48.872 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup config 2024-03-26T04:19:48.872 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.872 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.872 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:19:48.872 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.872 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:19:48.872 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.872 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:19:48.872 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:19:48.872 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.872 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:19:48.872 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.873 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:19:48.873 INFO:tasks.workunit.client.0.smithi016.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:19:48.873 INFO:tasks.workunit.client.0.smithi016.stderr:++ echo handler_error 2024-03-26T04:19:48.873 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:19:48.873 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:19:48.873 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:19:48.873 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.873 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:19:48.873 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.873 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:19:48.873 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:19:48.873 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.873 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:19:48.873 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.873 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:19:48.874 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.874 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:19:48.874 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.874 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:19:48.874 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup 2024-03-26T04:19:48.874 INFO:tasks.workunit.client.0.smithi016.stderr:+++++ readlink /proc/74980/exe 2024-03-26T04:19:48.874 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/locale 2024-03-26T04:19:48.874 INFO:tasks.workunit.client.0.smithi016.stderr:++++ basename /usr/bin/bash 2024-03-26T04:19:48.874 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_declare='declare -f' 2024-03-26T04:19:48.874 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_opt=-f 2024-03-26T04:19:48.874 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export which_declare 2024-03-26T04:19:48.874 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export -f which 2024-03-26T04:19:48.874 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset i 2024-03-26T04:19:48.874 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset -f pathmunge 2024-03-26T04:19:48.875 INFO:tasks.workunit.client.0.smithi016.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:19:48.875 INFO:tasks.workunit.client.0.smithi016.stderr:+ export PATH 2024-03-26T04:19:48.875 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:19:48.875 INFO:tasks.workunit.client.0.smithi016.stderr:+ unset rc 2024-03-26T04:19:48.875 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.875 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:19:48.875 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.875 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:19:48.875 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:19:48.875 INFO:tasks.workunit.client.0.smithi016.stderr:+++++ readlink /proc/75055/exe 2024-03-26T04:19:48.875 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:19:48.875 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2024-03-26T04:19:48.875 INFO:tasks.workunit.client.0.smithi016.stderr:++++ basename /usr/bin/bash 2024-03-26T04:19:48.875 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_declare='declare -f' 2024-03-26T04:19:48.876 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_opt=-f 2024-03-26T04:19:48.876 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export which_declare 2024-03-26T04:19:48.876 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export -f which 2024-03-26T04:19:48.876 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset i 2024-03-26T04:19:48.876 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset -f pathmunge 2024-03-26T04:19:48.876 INFO:tasks.workunit.client.0.smithi016.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-03-26T04:19:48.876 INFO:tasks.workunit.client.0.smithi016.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:19:48.876 INFO:tasks.workunit.client.0.smithi016.stderr:+ export PATH 2024-03-26T04:19:48.876 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:19:48.876 INFO:tasks.workunit.client.0.smithi016.stderr:+ unset rc 2024-03-26T04:19:48.876 INFO:tasks.workunit.client.0.smithi016.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:19:48.876 INFO:tasks.workunit.client.0.smithi016.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:19:48.876 INFO:tasks.workunit.client.0.smithi016.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:19:48.876 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.877 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:19:48.877 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.877 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:19:48.877 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup config 2024-03-26T04:19:48.877 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.877 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph_test_neorados_cls 2024-03-26T04:19:48.877 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.877 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:19:48.877 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.877 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:19:48.877 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.877 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:19:48.877 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:19:48.877 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.878 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:19:48.878 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.878 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:19:48.878 INFO:tasks.workunit.client.0.smithi016.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:19:48.878 INFO:tasks.workunit.client.0.smithi016.stderr:+ sed 's/^/ cls: /' 2024-03-26T04:19:48.878 INFO:tasks.workunit.client.0.smithi016.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-03-26T04:19:48.878 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:19:48.878 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:19:48.878 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:19:48.878 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.878 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:19:48.878 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.878 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:19:48.878 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:19:48.879 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.879 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:19:48.879 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.879 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:19:48.879 INFO:tasks.workunit.client.0.smithi016.stderr:+ tee ceph_test_neorados_cls.log 2024-03-26T04:19:48.879 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.879 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:19:48.879 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.879 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:19:48.879 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup 2024-03-26T04:19:48.879 INFO:tasks.workunit.client.0.smithi016.stderr:+++++ readlink /proc/75016/exe 2024-03-26T04:19:48.879 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/locale 2024-03-26T04:19:48.879 INFO:tasks.workunit.client.0.smithi016.stderr:++ awk '{print $1}' 2024-03-26T04:19:48.879 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.879 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:19:48.880 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.880 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:19:48.880 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:19:48.880 INFO:tasks.workunit.client.0.smithi016.stderr:++++ basename /usr/bin/bash 2024-03-26T04:19:48.880 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:19:48.880 INFO:tasks.workunit.client.0.smithi016.stdout:test handler_error on pid 75209 2024-03-26T04:19:48.886 INFO:tasks.workunit.client.0.smithi016.stderr:+ ff=handler_error 2024-03-26T04:19:48.886 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=75209 2024-03-26T04:19:48.886 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'test handler_error on pid 75209' 2024-03-26T04:19:48.886 INFO:tasks.workunit.client.0.smithi016.stderr:+ pids[$f]=75209 2024-03-26T04:19:48.886 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T04:19:48.886 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:19:48.886 INFO:tasks.workunit.client.0.smithi016.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:19:48.886 INFO:tasks.workunit.client.0.smithi016.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:19:48.887 INFO:tasks.workunit.client.0.smithi016.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:19:48.887 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.887 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:19:48.887 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.887 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:19:48.887 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup config 2024-03-26T04:19:48.887 INFO:tasks.workunit.client.0.smithi016.stderr:++ printf %25s io 2024-03-26T04:19:48.887 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.887 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.887 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:19:48.887 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.887 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:19:48.887 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.887 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:19:48.887 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:19:48.888 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.888 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:19:48.888 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.888 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:19:48.888 INFO:tasks.workunit.client.0.smithi016.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:19:48.888 INFO:tasks.workunit.client.0.smithi016.stderr:+ r=' io' 2024-03-26T04:19:48.888 INFO:tasks.workunit.client.0.smithi016.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2024-03-26T04:19:48.888 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_declare='declare -f' 2024-03-26T04:19:48.888 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_opt=-f 2024-03-26T04:19:48.888 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export which_declare 2024-03-26T04:19:48.888 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export -f which 2024-03-26T04:19:48.888 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset i 2024-03-26T04:19:48.888 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset -f pathmunge 2024-03-26T04:19:48.888 INFO:tasks.workunit.client.0.smithi016.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:19:48.889 INFO:tasks.workunit.client.0.smithi016.stderr:+ export PATH 2024-03-26T04:19:48.889 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:19:48.889 INFO:tasks.workunit.client.0.smithi016.stderr:+ unset rc 2024-03-26T04:19:48.889 INFO:tasks.workunit.client.0.smithi016.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-03-26T04:19:48.889 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2024-03-26T04:19:48.889 INFO:tasks.workunit.client.0.smithi016.stderr:+++++ readlink /proc/75075/exe 2024-03-26T04:19:48.889 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:19:48.889 INFO:tasks.workunit.client.0.smithi016.stderr:+ . /etc/bashrc 2024-03-26T04:19:48.889 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -z '' ']' 2024-03-26T04:19:48.889 INFO:tasks.workunit.client.0.smithi016.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:19:48.889 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.889 INFO:tasks.workunit.client.0.smithi016.stderr:++ shopt -q login_shell 2024-03-26T04:19:48.889 INFO:tasks.workunit.client.0.smithi016.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-03-26T04:19:48.889 INFO:tasks.workunit.client.0.smithi016.stderr:+++ umask 2024-03-26T04:19:48.889 INFO:tasks.workunit.client.0.smithi016.stderr:++ echo io 2024-03-26T04:19:48.890 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:19:48.890 INFO:tasks.workunit.client.0.smithi016.stderr:++ SHELL=/bin/bash 2024-03-26T04:19:48.890 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.890 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:19:48.890 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.890 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:19:48.890 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:19:48.890 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.890 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:19:48.890 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.890 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:19:48.890 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.890 INFO:tasks.workunit.client.0.smithi016.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.891 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.891 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:19:48.891 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.891 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:19:48.891 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.891 INFO:tasks.workunit.client.0.smithi016.stderr:++ awk '{print $1}' 2024-03-26T04:19:48.891 INFO:tasks.workunit.client.0.smithi016.stderr:++++ basename /usr/bin/bash 2024-03-26T04:19:48.891 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_declare='declare -f' 2024-03-26T04:19:48.891 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_opt=-f 2024-03-26T04:19:48.891 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export which_declare 2024-03-26T04:19:48.891 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export -f which 2024-03-26T04:19:48.891 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset i 2024-03-26T04:19:48.891 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset -f pathmunge 2024-03-26T04:19:48.891 INFO:tasks.workunit.client.0.smithi016.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:19:48.891 INFO:tasks.workunit.client.0.smithi016.stderr:+ export PATH 2024-03-26T04:19:48.892 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:19:48.892 INFO:tasks.workunit.client.0.smithi016.stderr:+ unset rc 2024-03-26T04:19:48.892 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph_test_neorados_cmd 2024-03-26T04:19:48.892 INFO:tasks.workunit.client.0.smithi016.stderr:+ sed 's/^/ cmd: /' 2024-03-26T04:19:48.892 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:19:48.892 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:19:48.892 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:19:48.892 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.892 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:19:48.892 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.892 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:19:48.892 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:19:48.892 INFO:tasks.workunit.client.0.smithi016.stderr:+++ return 2024-03-26T04:19:48.892 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.892 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:19:48.893 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.893 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:19:48.893 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export S_COLORS=auto 2024-03-26T04:19:48.893 INFO:tasks.workunit.client.0.smithi016.stderr:+++ S_COLORS=auto 2024-03-26T04:19:48.893 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.893 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:19:48.893 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.893 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:19:48.893 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.893 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:19:48.893 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:19:48.893 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:19:48.893 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.893 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:19:48.893 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.894 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:19:48.894 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:19:48.894 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.894 INFO:tasks.workunit.client.0.smithi016.stdout:test io on pid 75275 2024-03-26T04:19:48.895 INFO:tasks.workunit.client.0.smithi016.stderr:+ ff=io 2024-03-26T04:19:48.895 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=75275 2024-03-26T04:19:48.895 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'test io on pid 75275' 2024-03-26T04:19:48.895 INFO:tasks.workunit.client.0.smithi016.stderr:+ pids[$f]=75275 2024-03-26T04:19:48.896 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T04:19:48.896 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:19:48.896 INFO:tasks.workunit.client.0.smithi016.stderr:+ tee ceph_test_neorados_cmd.log 2024-03-26T04:19:48.896 INFO:tasks.workunit.client.0.smithi016.stderr:++ printf %25s ec_io 2024-03-26T04:19:48.896 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:19:48.896 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:19:48.896 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:19:48.896 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.896 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:19:48.896 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.896 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:19:48.896 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:19:48.896 INFO:tasks.workunit.client.0.smithi016.stderr:+ r=' ec_io' 2024-03-26T04:19:48.896 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.897 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:19:48.897 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.897 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:19:48.897 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.897 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:19:48.897 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.897 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:19:48.897 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup 2024-03-26T04:19:48.897 INFO:tasks.workunit.client.0.smithi016.stderr:++ echo ec_io 2024-03-26T04:19:48.897 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/locale 2024-03-26T04:19:48.897 INFO:tasks.workunit.client.0.smithi016.stderr:++ awk '{print $1}' 2024-03-26T04:19:48.897 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.897 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:19:48.897 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.898 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:19:48.898 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:19:48.898 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:19:48.898 INFO:tasks.workunit.client.0.smithi016.stdout:test ec_io on pid 75290 2024-03-26T04:19:48.899 INFO:tasks.workunit.client.0.smithi016.stderr:+ ff=ec_io 2024-03-26T04:19:48.900 INFO:tasks.workunit.client.0.smithi016.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:19:48.900 INFO:tasks.workunit.client.0.smithi016.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:19:48.900 INFO:tasks.workunit.client.0.smithi016.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:19:48.900 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.900 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:19:48.900 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.900 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:19:48.900 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=75290 2024-03-26T04:19:48.900 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup config 2024-03-26T04:19:48.900 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'test ec_io on pid 75290' 2024-03-26T04:19:48.900 INFO:tasks.workunit.client.0.smithi016.stderr:+ pids[$f]=75290 2024-03-26T04:19:48.900 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.900 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T04:19:48.900 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:19:48.901 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.901 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:19:48.901 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.901 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:19:48.901 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.901 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:19:48.901 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:19:48.901 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.901 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:19:48.901 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.901 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:19:48.901 INFO:tasks.workunit.client.0.smithi016.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:19:48.901 INFO:tasks.workunit.client.0.smithi016.stderr:++ printf %25s list 2024-03-26T04:19:48.901 INFO:tasks.workunit.client.0.smithi016.stderr:+ r=' list' 2024-03-26T04:19:48.901 INFO:tasks.workunit.client.0.smithi016.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_io 2>&1 | tee ceph_test_neorados_ec_io.log | sed "s/^/ ec_io: /"' 2024-03-26T04:19:48.902 INFO:tasks.workunit.client.0.smithi016.stderr:++ awk '{print $1}' 2024-03-26T04:19:48.902 INFO:tasks.workunit.client.0.smithi016.stderr:++ echo list 2024-03-26T04:19:48.902 INFO:tasks.workunit.client.0.smithi016.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2024-03-26T04:19:48.902 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:19:48.902 INFO:tasks.workunit.client.0.smithi016.stderr:+ . /etc/bashrc 2024-03-26T04:19:48.902 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -z '' ']' 2024-03-26T04:19:48.902 INFO:tasks.workunit.client.0.smithi016.stderr:+ ff=list 2024-03-26T04:19:48.902 INFO:tasks.workunit.client.0.smithi016.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:19:48.902 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.902 INFO:tasks.workunit.client.0.smithi016.stderr:++ shopt -q login_shell 2024-03-26T04:19:48.902 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=75303 2024-03-26T04:19:48.903 INFO:tasks.workunit.client.0.smithi016.stdout:test list on pid 75303 2024-03-26T04:19:48.911 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'test list on pid 75303' 2024-03-26T04:19:48.911 INFO:tasks.workunit.client.0.smithi016.stderr:+ pids[$f]=75303 2024-03-26T04:19:48.911 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T04:19:48.911 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:19:48.911 INFO:tasks.workunit.client.0.smithi016.stderr:+++++ readlink /proc/75209/exe 2024-03-26T04:19:48.911 INFO:tasks.workunit.client.0.smithi016.stderr:+++ umask 2024-03-26T04:19:48.911 INFO:tasks.workunit.client.0.smithi016.stderr:++ printf %25s ec_list 2024-03-26T04:19:48.911 INFO:tasks.workunit.client.0.smithi016.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2024-03-26T04:19:48.911 INFO:tasks.workunit.client.0.smithi016.stderr:++++ basename /usr/bin/bash 2024-03-26T04:19:48.911 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:19:48.911 INFO:tasks.workunit.client.0.smithi016.stderr:++ SHELL=/bin/bash 2024-03-26T04:19:48.911 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.912 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:19:48.912 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.912 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:19:48.912 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:19:48.912 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.912 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:19:48.912 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.912 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:19:48.912 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.912 INFO:tasks.workunit.client.0.smithi016.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.912 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.912 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:19:48.912 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.912 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:19:48.913 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.913 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_declare='declare -f' 2024-03-26T04:19:48.913 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_opt=-f 2024-03-26T04:19:48.913 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export which_declare 2024-03-26T04:19:48.913 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export -f which 2024-03-26T04:19:48.913 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset i 2024-03-26T04:19:48.913 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset -f pathmunge 2024-03-26T04:19:48.913 INFO:tasks.workunit.client.0.smithi016.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:19:48.913 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:19:48.913 INFO:tasks.workunit.client.0.smithi016.stderr:+ export PATH 2024-03-26T04:19:48.913 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:19:48.913 INFO:tasks.workunit.client.0.smithi016.stderr:+ unset rc 2024-03-26T04:19:48.913 INFO:tasks.workunit.client.0.smithi016.stderr:+ r=' ec_list' 2024-03-26T04:19:48.913 INFO:tasks.workunit.client.0.smithi016.stderr:+ . /etc/bashrc 2024-03-26T04:19:48.913 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:19:48.914 INFO:tasks.workunit.client.0.smithi016.stderr:+ . /etc/bashrc 2024-03-26T04:19:48.914 INFO:tasks.workunit.client.0.smithi016.stderr:+ sed 's/^/ handler_error: /' 2024-03-26T04:19:48.914 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -z '' ']' 2024-03-26T04:19:48.914 INFO:tasks.workunit.client.0.smithi016.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:19:48.914 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -z '' ']' 2024-03-26T04:19:48.914 INFO:tasks.workunit.client.0.smithi016.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:19:48.914 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.914 INFO:tasks.workunit.client.0.smithi016.stderr:++ shopt -q login_shell 2024-03-26T04:19:48.914 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.914 INFO:tasks.workunit.client.0.smithi016.stderr:++ shopt -q login_shell 2024-03-26T04:19:48.914 INFO:tasks.workunit.client.0.smithi016.stderr:+++ umask 2024-03-26T04:19:48.914 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:19:48.914 INFO:tasks.workunit.client.0.smithi016.stderr:++ SHELL=/bin/bash 2024-03-26T04:19:48.914 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.915 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:19:48.915 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.915 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:19:48.915 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:19:48.915 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.915 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:19:48.915 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.915 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:19:48.915 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.915 INFO:tasks.workunit.client.0.smithi016.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.915 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.915 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:19:48.915 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.915 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:19:48.915 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.916 INFO:tasks.workunit.client.0.smithi016.stderr:+++ umask 2024-03-26T04:19:48.916 INFO:tasks.workunit.client.0.smithi016.stderr:+ tee ceph_test_neorados_handler_error.log 2024-03-26T04:19:48.916 INFO:tasks.workunit.client.0.smithi016.stderr:++ echo ec_list 2024-03-26T04:19:48.916 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:19:48.916 INFO:tasks.workunit.client.0.smithi016.stderr:++ SHELL=/bin/bash 2024-03-26T04:19:48.916 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.916 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:19:48.916 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph_test_neorados_handler_error 2024-03-26T04:19:48.916 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.916 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:19:48.916 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:19:48.916 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.916 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:19:48.916 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.917 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:19:48.917 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.917 INFO:tasks.workunit.client.0.smithi016.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.917 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.917 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:19:48.917 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.917 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:19:48.917 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.917 INFO:tasks.workunit.client.0.smithi016.stderr:++ awk '{print $1}' 2024-03-26T04:19:48.917 INFO:tasks.workunit.client.0.smithi016.stderr:+ ff=ec_list 2024-03-26T04:19:48.917 INFO:tasks.workunit.client.0.smithi016.stdout:test ec_list on pid 75328 2024-03-26T04:19:48.920 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=75328 2024-03-26T04:19:48.920 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'test ec_list on pid 75328' 2024-03-26T04:19:48.920 INFO:tasks.workunit.client.0.smithi016.stderr:+ pids[$f]=75328 2024-03-26T04:19:48.920 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T04:19:48.920 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:19:48.921 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:19:48.921 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:19:48.921 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:19:48.921 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.921 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:19:48.921 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.921 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:19:48.921 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:19:48.921 INFO:tasks.workunit.client.0.smithi016.stderr:+++ return 2024-03-26T04:19:48.921 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.921 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:19:48.921 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.921 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:19:48.921 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export S_COLORS=auto 2024-03-26T04:19:48.922 INFO:tasks.workunit.client.0.smithi016.stderr:+++ S_COLORS=auto 2024-03-26T04:19:48.922 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.922 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:19:48.922 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.922 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:19:48.922 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.922 INFO:tasks.workunit.client.0.smithi016.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_list 2>&1 | tee ceph_test_neorados_ec_list.log | sed "s/^/ ec_list: /"' 2024-03-26T04:19:48.922 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:19:48.922 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:19:48.922 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:19:48.922 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.922 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:19:48.922 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.922 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:19:48.923 INFO:tasks.workunit.client.0.smithi016.stderr:++ printf %25s misc 2024-03-26T04:19:48.923 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:19:48.923 INFO:tasks.workunit.client.0.smithi016.stderr:+++ return 2024-03-26T04:19:48.923 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.923 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:19:48.923 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.923 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:19:48.923 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export S_COLORS=auto 2024-03-26T04:19:48.923 INFO:tasks.workunit.client.0.smithi016.stderr:+++ S_COLORS=auto 2024-03-26T04:19:48.923 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.923 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:19:48.923 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.923 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:19:48.923 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.924 INFO:tasks.workunit.client.0.smithi016.stderr:+ r=' misc' 2024-03-26T04:19:48.924 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:19:48.924 INFO:tasks.workunit.client.0.smithi016.stderr:+ . /etc/bashrc 2024-03-26T04:19:48.924 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -z '' ']' 2024-03-26T04:19:48.924 INFO:tasks.workunit.client.0.smithi016.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:19:48.924 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.924 INFO:tasks.workunit.client.0.smithi016.stderr:++ shopt -q login_shell 2024-03-26T04:19:48.924 INFO:tasks.workunit.client.0.smithi016.stderr:+++ umask 2024-03-26T04:19:48.924 INFO:tasks.workunit.client.0.smithi016.stderr:++ awk '{print $1}' 2024-03-26T04:19:48.924 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:19:48.924 INFO:tasks.workunit.client.0.smithi016.stderr:++ SHELL=/bin/bash 2024-03-26T04:19:48.924 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.924 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:19:48.925 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.925 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:19:48.925 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:19:48.925 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.925 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:19:48.925 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.925 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:19:48.925 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.925 INFO:tasks.workunit.client.0.smithi016.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.925 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.925 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:19:48.925 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.925 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:19:48.925 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.926 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:19:48.926 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:19:48.926 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:19:48.926 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.926 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:19:48.926 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.926 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:19:48.926 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:19:48.926 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.926 INFO:tasks.workunit.client.0.smithi016.stderr:++ echo misc 2024-03-26T04:19:48.926 INFO:tasks.workunit.client.0.smithi016.stdout:test misc on pid 75363 2024-03-26T04:19:48.931 INFO:tasks.workunit.client.0.smithi016.stderr:+ ff=misc 2024-03-26T04:19:48.931 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=75363 2024-03-26T04:19:48.931 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'test misc on pid 75363' 2024-03-26T04:19:48.931 INFO:tasks.workunit.client.0.smithi016.stderr:+ pids[$f]=75363 2024-03-26T04:19:48.931 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T04:19:48.931 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:19:48.931 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:19:48.931 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:19:48.931 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:19:48.931 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.931 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:19:48.931 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.932 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:19:48.932 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:19:48.932 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.932 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:19:48.932 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:19:48.932 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:19:48.932 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.932 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:19:48.932 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.932 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:19:48.932 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:19:48.932 INFO:tasks.workunit.client.0.smithi016.stderr:+++ return 2024-03-26T04:19:48.932 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.932 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:19:48.933 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.933 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:19:48.933 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export S_COLORS=auto 2024-03-26T04:19:48.933 INFO:tasks.workunit.client.0.smithi016.stderr:+++ S_COLORS=auto 2024-03-26T04:19:48.933 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.933 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:19:48.933 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.933 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:19:48.933 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.933 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:19:48.933 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:19:48.933 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:19:48.933 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.933 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:19:48.934 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.934 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:19:48.934 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:19:48.934 INFO:tasks.workunit.client.0.smithi016.stderr:+++ return 2024-03-26T04:19:48.934 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.934 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:19:48.934 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.934 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:19:48.934 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export S_COLORS=auto 2024-03-26T04:19:48.934 INFO:tasks.workunit.client.0.smithi016.stderr:+++ S_COLORS=auto 2024-03-26T04:19:48.934 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.934 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:19:48.934 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.934 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:19:48.935 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.935 INFO:tasks.workunit.client.0.smithi016.stderr:++ printf %25s pool 2024-03-26T04:19:48.935 INFO:tasks.workunit.client.0.smithi016.stderr:+ r=' pool' 2024-03-26T04:19:48.935 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:19:48.935 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:19:48.935 INFO:tasks.workunit.client.0.smithi016.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2024-03-26T04:19:48.935 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:19:48.935 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.935 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:19:48.935 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.935 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:19:48.935 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:19:48.935 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.935 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:19:48.936 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.936 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:19:48.936 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.936 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:19:48.936 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.936 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:19:48.936 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup 2024-03-26T04:19:48.936 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/locale 2024-03-26T04:19:48.936 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:19:48.936 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:19:48.936 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:19:48.936 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.936 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:19:48.936 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.937 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:19:48.937 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:19:48.937 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.937 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:19:48.937 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.937 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:19:48.937 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.937 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:19:48.937 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.937 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:19:48.937 INFO:tasks.workunit.client.0.smithi016.stderr:++ echo pool 2024-03-26T04:19:48.937 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup 2024-03-26T04:19:48.937 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/locale 2024-03-26T04:19:48.937 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:19:48.938 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:19:48.938 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:19:48.938 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.938 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:19:48.938 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.938 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:19:48.938 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:19:48.938 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.938 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:19:48.938 INFO:tasks.workunit.client.0.smithi016.stderr:+ . /etc/bashrc 2024-03-26T04:19:48.938 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -z '' ']' 2024-03-26T04:19:48.938 INFO:tasks.workunit.client.0.smithi016.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:19:48.938 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.938 INFO:tasks.workunit.client.0.smithi016.stderr:++ shopt -q login_shell 2024-03-26T04:19:48.939 INFO:tasks.workunit.client.0.smithi016.stderr:+++ umask 2024-03-26T04:19:48.939 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:19:48.939 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:19:48.939 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:19:48.939 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:19:48.939 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.939 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:19:48.939 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.939 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:19:48.939 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:19:48.939 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:19:48.939 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:19:48.939 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.939 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.939 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:19:48.940 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:19:48.940 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.940 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:19:48.940 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.940 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:19:48.940 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:19:48.940 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.940 INFO:tasks.workunit.client.0.smithi016.stderr:++ awk '{print $1}' 2024-03-26T04:19:48.940 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.940 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:19:48.940 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.940 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:19:48.940 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup 2024-03-26T04:19:48.940 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.941 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:19:48.941 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:19:48.941 INFO:tasks.workunit.client.0.smithi016.stderr:++ SHELL=/bin/bash 2024-03-26T04:19:48.941 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/locale 2024-03-26T04:19:48.941 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.941 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:19:48.941 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:19:48.941 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.941 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:19:48.941 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.941 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:19:48.941 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:19:48.941 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.941 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:19:48.942 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.942 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:19:48.942 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.942 INFO:tasks.workunit.client.0.smithi016.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.942 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.942 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:19:48.942 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.942 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:19:48.942 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.942 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:19:48.942 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.942 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:19:48.942 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.942 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:19:48.943 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:19:48.943 INFO:tasks.workunit.client.0.smithi016.stderr:+ ff=pool 2024-03-26T04:19:48.943 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=75441 2024-03-26T04:19:48.943 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'test pool on pid 75441' 2024-03-26T04:19:48.943 INFO:tasks.workunit.client.0.smithi016.stdout:test pool on pid 75441 2024-03-26T04:19:48.946 INFO:tasks.workunit.client.0.smithi016.stderr:+ pids[$f]=75441 2024-03-26T04:19:48.947 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T04:19:48.947 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:19:48.947 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:19:48.947 INFO:tasks.workunit.client.0.smithi016.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:19:48.947 INFO:tasks.workunit.client.0.smithi016.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:19:48.947 INFO:tasks.workunit.client.0.smithi016.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:19:48.947 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.947 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:19:48.947 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.947 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:19:48.947 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup config 2024-03-26T04:19:48.947 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.947 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.947 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:19:48.948 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.948 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:19:48.948 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.948 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:19:48.948 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:19:48.948 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.948 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:19:48.948 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.948 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:19:48.948 INFO:tasks.workunit.client.0.smithi016.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:19:48.948 INFO:tasks.workunit.client.0.smithi016.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2024-03-26T04:19:48.948 INFO:tasks.workunit.client.0.smithi016.stderr:++ printf %25s read_operations 2024-03-26T04:19:48.948 INFO:tasks.workunit.client.0.smithi016.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:19:48.948 INFO:tasks.workunit.client.0.smithi016.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:19:48.948 INFO:tasks.workunit.client.0.smithi016.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:19:48.949 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.949 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:19:48.949 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.949 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:19:48.949 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup config 2024-03-26T04:19:48.949 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.949 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.949 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:19:48.949 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.949 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:19:48.949 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.949 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:19:48.949 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:19:48.949 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.950 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:19:48.950 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.950 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:19:48.950 INFO:tasks.workunit.client.0.smithi016.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:19:48.950 INFO:tasks.workunit.client.0.smithi016.stderr:+ r=' read_operations' 2024-03-26T04:19:48.950 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.950 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:19:48.950 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.950 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:19:48.950 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:19:48.950 INFO:tasks.workunit.client.0.smithi016.stderr:+++++ readlink /proc/75303/exe 2024-03-26T04:19:48.950 INFO:tasks.workunit.client.0.smithi016.stderr:++ echo read_operations 2024-03-26T04:19:48.950 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:19:48.950 INFO:tasks.workunit.client.0.smithi016.stderr:+ . /etc/bashrc 2024-03-26T04:19:48.951 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:19:48.951 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:19:48.951 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:19:48.951 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:19:48.951 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:19:48.951 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:19:48.951 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.951 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:19:48.951 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.951 INFO:tasks.workunit.client.0.smithi016.stderr:+++++ readlink /proc/75290/exe 2024-03-26T04:19:48.951 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.951 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:19:48.951 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:19:48.951 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.951 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:19:48.952 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -z '' ']' 2024-03-26T04:19:48.952 INFO:tasks.workunit.client.0.smithi016.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:19:48.952 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.952 INFO:tasks.workunit.client.0.smithi016.stderr:++ shopt -q login_shell 2024-03-26T04:19:48.952 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:19:48.952 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:19:48.952 INFO:tasks.workunit.client.0.smithi016.stderr:+++ return 2024-03-26T04:19:48.952 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.952 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.952 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:19:48.952 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:19:48.952 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.952 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.952 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:19:48.952 INFO:tasks.workunit.client.0.smithi016.stderr:++ awk '{print $1}' 2024-03-26T04:19:48.953 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:19:48.953 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export S_COLORS=auto 2024-03-26T04:19:48.953 INFO:tasks.workunit.client.0.smithi016.stderr:+++ S_COLORS=auto 2024-03-26T04:19:48.953 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.953 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:19:48.953 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.953 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:19:48.953 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.953 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.953 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:19:48.953 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.953 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:19:48.953 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup 2024-03-26T04:19:48.953 INFO:tasks.workunit.client.0.smithi016.stderr:+++ umask 2024-03-26T04:19:48.953 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:19:48.954 INFO:tasks.workunit.client.0.smithi016.stderr:++ SHELL=/bin/bash 2024-03-26T04:19:48.954 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.954 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/locale 2024-03-26T04:19:48.954 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:19:48.954 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.954 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:19:48.954 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:19:48.954 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.954 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:19:48.954 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.954 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:19:48.954 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.954 INFO:tasks.workunit.client.0.smithi016.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.954 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.954 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:19:48.955 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.955 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:19:48.955 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.955 INFO:tasks.workunit.client.0.smithi016.stderr:++++ basename /usr/bin/bash 2024-03-26T04:19:48.955 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_declare='declare -f' 2024-03-26T04:19:48.955 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_opt=-f 2024-03-26T04:19:48.955 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export which_declare 2024-03-26T04:19:48.955 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export -f which 2024-03-26T04:19:48.955 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:19:48.955 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset i 2024-03-26T04:19:48.955 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset -f pathmunge 2024-03-26T04:19:48.955 INFO:tasks.workunit.client.0.smithi016.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:19:48.955 INFO:tasks.workunit.client.0.smithi016.stderr:+ export PATH 2024-03-26T04:19:48.955 INFO:tasks.workunit.client.0.smithi016.stderr:++++ basename /usr/bin/bash 2024-03-26T04:19:48.956 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_declare='declare -f' 2024-03-26T04:19:48.956 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_opt=-f 2024-03-26T04:19:48.956 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export which_declare 2024-03-26T04:19:48.956 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export -f which 2024-03-26T04:19:48.956 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset i 2024-03-26T04:19:48.956 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset -f pathmunge 2024-03-26T04:19:48.956 INFO:tasks.workunit.client.0.smithi016.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:19:48.956 INFO:tasks.workunit.client.0.smithi016.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:19:48.956 INFO:tasks.workunit.client.0.smithi016.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:19:48.956 INFO:tasks.workunit.client.0.smithi016.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:19:48.956 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.956 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:19:48.956 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:19:48.956 INFO:tasks.workunit.client.0.smithi016.stderr:+ unset rc 2024-03-26T04:19:48.956 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.957 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:19:48.957 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup config 2024-03-26T04:19:48.957 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.957 INFO:tasks.workunit.client.0.smithi016.stdout:test read_operations on pid 75484 2024-03-26T04:19:48.960 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.960 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:19:48.960 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.960 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:19:48.960 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:19:48.960 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.960 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:19:48.960 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:19:48.960 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:19:48.960 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:19:48.961 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.961 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.961 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:19:48.961 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:19:48.961 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.961 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:19:48.961 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.961 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:19:48.961 INFO:tasks.workunit.client.0.smithi016.stderr:+ ff=read_operations 2024-03-26T04:19:48.961 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:19:48.961 INFO:tasks.workunit.client.0.smithi016.stderr:+++ return 2024-03-26T04:19:48.961 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.961 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.961 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:19:48.961 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.962 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:19:48.962 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:19:48.962 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export S_COLORS=auto 2024-03-26T04:19:48.962 INFO:tasks.workunit.client.0.smithi016.stderr:+++ S_COLORS=auto 2024-03-26T04:19:48.962 INFO:tasks.workunit.client.0.smithi016.stderr:+ export PATH 2024-03-26T04:19:48.962 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.962 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.962 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:19:48.962 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:19:48.962 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.962 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:19:48.962 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:19:48.962 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:19:48.962 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=75484 2024-03-26T04:19:48.963 INFO:tasks.workunit.client.0.smithi016.stderr:+ unset rc 2024-03-26T04:19:48.963 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'test read_operations on pid 75484' 2024-03-26T04:19:48.963 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.963 INFO:tasks.workunit.client.0.smithi016.stderr:+ pids[$f]=75484 2024-03-26T04:19:48.963 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T04:19:48.963 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:19:48.963 INFO:tasks.workunit.client.0.smithi016.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2024-03-26T04:19:48.963 INFO:tasks.workunit.client.0.smithi016.stderr:+ sed 's/^/ list: /' 2024-03-26T04:19:48.963 INFO:tasks.workunit.client.0.smithi016.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:19:48.963 INFO:tasks.workunit.client.0.smithi016.stderr:++ printf %25s snapshots 2024-03-26T04:19:48.963 INFO:tasks.workunit.client.0.smithi016.stderr:+ tee ceph_test_neorados_list.log 2024-03-26T04:19:48.963 INFO:tasks.workunit.client.0.smithi016.stderr:+ r=' snapshots' 2024-03-26T04:19:48.963 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:19:48.963 INFO:tasks.workunit.client.0.smithi016.stderr:+ tee ceph_test_neorados_ec_io.log 2024-03-26T04:19:48.964 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:19:48.964 INFO:tasks.workunit.client.0.smithi016.stderr:+ . /etc/bashrc 2024-03-26T04:19:48.964 INFO:tasks.workunit.client.0.smithi016.stderr:+ sed 's/^/ ec_io: /' 2024-03-26T04:19:48.964 INFO:tasks.workunit.client.0.smithi016.stderr:+++++ readlink /proc/75275/exe 2024-03-26T04:19:48.964 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -z '' ']' 2024-03-26T04:19:48.964 INFO:tasks.workunit.client.0.smithi016.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:19:48.964 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.964 INFO:tasks.workunit.client.0.smithi016.stderr:++ shopt -q login_shell 2024-03-26T04:19:48.964 INFO:tasks.workunit.client.0.smithi016.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:19:48.964 INFO:tasks.workunit.client.0.smithi016.stderr:+++ umask 2024-03-26T04:19:48.964 INFO:tasks.workunit.client.0.smithi016.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:19:48.964 INFO:tasks.workunit.client.0.smithi016.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:19:48.964 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.964 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:19:48.964 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.965 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:19:48.965 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup config 2024-03-26T04:19:48.965 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.965 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:19:48.965 INFO:tasks.workunit.client.0.smithi016.stderr:++ SHELL=/bin/bash 2024-03-26T04:19:48.965 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.965 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:19:48.965 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.965 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:19:48.965 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.965 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.965 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:19:48.965 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:19:48.965 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.966 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:19:48.966 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:19:48.966 INFO:tasks.workunit.client.0.smithi016.stderr:++ awk '{print $1}' 2024-03-26T04:19:48.966 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.966 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:19:48.966 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.966 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:19:48.966 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:19:48.966 INFO:tasks.workunit.client.0.smithi016.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:19:48.966 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.966 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:19:48.966 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.966 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:19:48.966 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.967 INFO:tasks.workunit.client.0.smithi016.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.967 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.967 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:19:48.967 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.967 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:19:48.967 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.967 INFO:tasks.workunit.client.0.smithi016.stderr:++++ basename /usr/bin/bash 2024-03-26T04:19:48.967 INFO:tasks.workunit.client.0.smithi016.stderr:++ echo snapshots 2024-03-26T04:19:48.967 INFO:tasks.workunit.client.0.smithi016.stderr:+++++ readlink /proc/75328/exe 2024-03-26T04:19:48.967 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph_test_neorados_list 2024-03-26T04:19:48.967 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_declare='declare -f' 2024-03-26T04:19:48.967 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_opt=-f 2024-03-26T04:19:48.967 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export which_declare 2024-03-26T04:19:48.967 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export -f which 2024-03-26T04:19:48.967 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset i 2024-03-26T04:19:48.968 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset -f pathmunge 2024-03-26T04:19:48.968 INFO:tasks.workunit.client.0.smithi016.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:19:48.968 INFO:tasks.workunit.client.0.smithi016.stderr:+ export PATH 2024-03-26T04:19:48.968 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:19:48.968 INFO:tasks.workunit.client.0.smithi016.stderr:+ unset rc 2024-03-26T04:19:48.968 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph_test_neorados_ec_io 2024-03-26T04:19:48.968 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:19:48.968 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:19:48.968 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:19:48.968 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.968 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:19:48.968 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.968 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:19:48.968 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:19:48.969 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.969 INFO:tasks.workunit.client.0.smithi016.stderr:+ sed 's/^/ io: /' 2024-03-26T04:19:48.969 INFO:tasks.workunit.client.0.smithi016.stderr:+ tee ceph_test_neorados_io.log 2024-03-26T04:19:48.969 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph_test_neorados_io 2024-03-26T04:19:48.969 INFO:tasks.workunit.client.0.smithi016.stderr:+ ff=snapshots 2024-03-26T04:19:48.969 INFO:tasks.workunit.client.0.smithi016.stdout:test snapshots on pid 75521 2024-03-26T04:19:48.971 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=75521 2024-03-26T04:19:48.971 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'test snapshots on pid 75521' 2024-03-26T04:19:48.971 INFO:tasks.workunit.client.0.smithi016.stderr:+ pids[$f]=75521 2024-03-26T04:19:48.971 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T04:19:48.971 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:19:48.971 INFO:tasks.workunit.client.0.smithi016.stderr:++ printf %25s watch_notify 2024-03-26T04:19:48.972 INFO:tasks.workunit.client.0.smithi016.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2024-03-26T04:19:48.972 INFO:tasks.workunit.client.0.smithi016.stderr:+ r=' watch_notify' 2024-03-26T04:19:48.972 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:19:48.972 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:19:48.972 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:19:48.972 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.972 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:19:48.972 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.972 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:19:48.972 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:19:48.972 INFO:tasks.workunit.client.0.smithi016.stderr:+++ return 2024-03-26T04:19:48.972 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.972 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:19:48.972 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.973 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:19:48.973 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export S_COLORS=auto 2024-03-26T04:19:48.973 INFO:tasks.workunit.client.0.smithi016.stderr:+++ S_COLORS=auto 2024-03-26T04:19:48.973 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.973 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:19:48.973 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.973 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:19:48.973 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.973 INFO:tasks.workunit.client.0.smithi016.stderr:++ awk '{print $1}' 2024-03-26T04:19:48.973 INFO:tasks.workunit.client.0.smithi016.stderr:++++ basename /usr/bin/bash 2024-03-26T04:19:48.973 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:19:48.973 INFO:tasks.workunit.client.0.smithi016.stderr:++ echo watch_notify 2024-03-26T04:19:48.973 INFO:tasks.workunit.client.0.smithi016.stderr:+ . /etc/bashrc 2024-03-26T04:19:48.973 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -z '' ']' 2024-03-26T04:19:48.973 INFO:tasks.workunit.client.0.smithi016.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:19:48.974 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.974 INFO:tasks.workunit.client.0.smithi016.stderr:++ shopt -q login_shell 2024-03-26T04:19:48.974 INFO:tasks.workunit.client.0.smithi016.stderr:+++ umask 2024-03-26T04:19:48.974 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_declare='declare -f' 2024-03-26T04:19:48.974 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_opt=-f 2024-03-26T04:19:48.974 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export which_declare 2024-03-26T04:19:48.974 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export -f which 2024-03-26T04:19:48.974 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset i 2024-03-26T04:19:48.974 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset -f pathmunge 2024-03-26T04:19:48.974 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:19:48.974 INFO:tasks.workunit.client.0.smithi016.stderr:++ SHELL=/bin/bash 2024-03-26T04:19:48.974 INFO:tasks.workunit.client.0.smithi016.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:19:48.974 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.974 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:19:48.975 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.975 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:19:48.975 INFO:tasks.workunit.client.0.smithi016.stderr:+ export PATH 2024-03-26T04:19:48.975 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:19:48.975 INFO:tasks.workunit.client.0.smithi016.stderr:+ unset rc 2024-03-26T04:19:48.975 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:19:48.975 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.975 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:19:48.975 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.975 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:19:48.975 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.975 INFO:tasks.workunit.client.0.smithi016.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.975 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.975 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:19:48.975 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.976 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:19:48.976 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.976 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph_test_neorados_ec_list 2024-03-26T04:19:48.976 INFO:tasks.workunit.client.0.smithi016.stderr:+ tee ceph_test_neorados_ec_list.log 2024-03-26T04:19:48.976 INFO:tasks.workunit.client.0.smithi016.stderr:+ sed 's/^/ ec_list: /' 2024-03-26T04:19:48.976 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:19:48.976 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:19:48.976 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:19:48.976 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.976 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:19:48.976 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.976 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:19:48.976 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:19:48.976 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.976 INFO:tasks.workunit.client.0.smithi016.stderr:+ ff=watch_notify 2024-03-26T04:19:48.977 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=75535 2024-03-26T04:19:48.977 INFO:tasks.workunit.client.0.smithi016.stdout:test watch_notify on pid 75535 2024-03-26T04:19:48.983 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'test watch_notify on pid 75535' 2024-03-26T04:19:48.983 INFO:tasks.workunit.client.0.smithi016.stderr:+ pids[$f]=75535 2024-03-26T04:19:48.983 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-03-26T04:19:48.983 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:19:48.983 INFO:tasks.workunit.client.0.smithi016.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2024-03-26T04:19:48.983 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:19:48.983 INFO:tasks.workunit.client.0.smithi016.stderr:++ printf %25s write_operations 2024-03-26T04:19:48.983 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:19:48.984 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:19:48.984 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.984 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:19:48.984 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.984 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:19:48.984 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:19:48.984 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.984 INFO:tasks.workunit.client.0.smithi016.stderr:+ r=' write_operations' 2024-03-26T04:19:48.984 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:19:48.984 INFO:tasks.workunit.client.0.smithi016.stderr:+ . /etc/bashrc 2024-03-26T04:19:48.984 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -z '' ']' 2024-03-26T04:19:48.984 INFO:tasks.workunit.client.0.smithi016.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:19:48.984 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.984 INFO:tasks.workunit.client.0.smithi016.stderr:++ shopt -q login_shell 2024-03-26T04:19:48.984 INFO:tasks.workunit.client.0.smithi016.stderr:+++ umask 2024-03-26T04:19:48.985 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:19:48.985 INFO:tasks.workunit.client.0.smithi016.stderr:++ SHELL=/bin/bash 2024-03-26T04:19:48.985 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.985 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:19:48.985 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.985 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:19:48.985 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:19:48.985 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.985 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:19:48.985 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.985 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:19:48.985 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.985 INFO:tasks.workunit.client.0.smithi016.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:48.985 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.986 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:19:48.986 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.986 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:19:48.986 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.986 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:19:48.986 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:19:48.986 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:19:48.986 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.986 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:19:48.986 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.986 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:19:48.986 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:19:48.986 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.986 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:19:48.986 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.987 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:19:48.987 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.987 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:19:48.987 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.987 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:19:48.987 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup 2024-03-26T04:19:48.987 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/locale 2024-03-26T04:19:48.987 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:19:48.987 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:19:48.987 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:19:48.987 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.987 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:19:48.987 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.987 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:19:48.988 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:19:48.988 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.988 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:19:48.988 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.988 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:19:48.988 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.988 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:19:48.988 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.988 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:19:48.988 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup 2024-03-26T04:19:48.988 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/locale 2024-03-26T04:19:48.988 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.988 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:19:48.988 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:19:48.988 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.989 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:19:48.989 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:19:48.989 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:19:48.989 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:19:48.989 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.989 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:19:48.989 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.989 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:19:48.989 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:19:48.989 INFO:tasks.workunit.client.0.smithi016.stderr:+++ return 2024-03-26T04:19:48.989 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.989 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:19:48.989 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.989 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:19:48.989 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export S_COLORS=auto 2024-03-26T04:19:48.990 INFO:tasks.workunit.client.0.smithi016.stderr:+++ S_COLORS=auto 2024-03-26T04:19:48.990 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.990 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:19:48.990 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.990 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:19:48.990 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.990 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.990 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:19:48.990 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.990 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:19:48.990 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:19:48.990 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:19:48.990 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:19:48.990 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:19:48.991 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:19:48.991 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:19:48.991 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.991 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:19:48.991 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.991 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:19:48.991 INFO:tasks.workunit.client.0.smithi016.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:19:48.991 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:19:48.991 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.991 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:19:48.991 INFO:tasks.workunit.client.0.smithi016.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:19:48.991 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.991 INFO:tasks.workunit.client.0.smithi016.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:19:48.991 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:19:48.991 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.992 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:19:48.992 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.992 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:19:48.992 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.992 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup config 2024-03-26T04:19:48.992 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:19:48.992 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.992 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:19:48.992 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.992 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup 2024-03-26T04:19:48.992 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.992 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:19:48.992 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.992 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:19:48.992 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.993 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:19:48.993 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:19:48.993 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.993 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:19:48.993 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.993 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:19:48.993 INFO:tasks.workunit.client.0.smithi016.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:19:48.993 INFO:tasks.workunit.client.0.smithi016.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:19:48.993 INFO:tasks.workunit.client.0.smithi016.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:19:48.993 INFO:tasks.workunit.client.0.smithi016.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:19:48.993 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/locale 2024-03-26T04:19:48.993 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.993 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:19:48.993 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.993 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:19:48.994 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup config 2024-03-26T04:19:48.994 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.994 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.994 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:19:48.994 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.994 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:19:48.994 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.994 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:19:48.994 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:19:48.994 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.994 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:19:48.994 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.994 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:19:48.994 INFO:tasks.workunit.client.0.smithi016.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:19:48.995 INFO:tasks.workunit.client.0.smithi016.stderr:+++++ readlink /proc/75484/exe 2024-03-26T04:19:48.995 INFO:tasks.workunit.client.0.smithi016.stderr:+++++ readlink /proc/75363/exe 2024-03-26T04:19:48.995 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.995 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:19:48.995 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.995 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:19:48.995 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:19:48.995 INFO:tasks.workunit.client.0.smithi016.stderr:++++ basename /usr/bin/bash 2024-03-26T04:19:48.995 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:19:48.995 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:19:48.995 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:19:48.995 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:19:48.995 INFO:tasks.workunit.client.0.smithi016.stderr:++++ basename /usr/bin/bash 2024-03-26T04:19:48.995 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.996 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:19:48.996 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.996 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:19:48.996 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:19:48.996 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:48.996 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_declare='declare -f' 2024-03-26T04:19:48.996 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_opt=-f 2024-03-26T04:19:48.996 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export which_declare 2024-03-26T04:19:48.996 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export -f which 2024-03-26T04:19:48.996 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset i 2024-03-26T04:19:48.996 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset -f pathmunge 2024-03-26T04:19:48.996 INFO:tasks.workunit.client.0.smithi016.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:19:48.996 INFO:tasks.workunit.client.0.smithi016.stderr:+ export PATH 2024-03-26T04:19:48.996 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:19:48.996 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_declare='declare -f' 2024-03-26T04:19:48.997 INFO:tasks.workunit.client.0.smithi016.stderr:+ unset rc 2024-03-26T04:19:48.997 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_opt=-f 2024-03-26T04:19:48.997 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export which_declare 2024-03-26T04:19:48.997 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export -f which 2024-03-26T04:19:48.997 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset i 2024-03-26T04:19:48.997 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset -f pathmunge 2024-03-26T04:19:48.997 INFO:tasks.workunit.client.0.smithi016.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:19:48.997 INFO:tasks.workunit.client.0.smithi016.stderr:+ export PATH 2024-03-26T04:19:48.997 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:19:48.997 INFO:tasks.workunit.client.0.smithi016.stderr:+ unset rc 2024-03-26T04:19:48.997 INFO:tasks.workunit.client.0.smithi016.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:19:48.997 INFO:tasks.workunit.client.0.smithi016.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:19:48.997 INFO:tasks.workunit.client.0.smithi016.stderr:+ sed 's/^/ read_operations: /' 2024-03-26T04:19:48.997 INFO:tasks.workunit.client.0.smithi016.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:19:48.998 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:48.998 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:19:48.998 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.998 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:19:48.998 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup config 2024-03-26T04:19:48.998 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:48.998 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:48.998 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph_test_neorados_misc 2024-03-26T04:19:48.998 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph_test_neorados_read_operations 2024-03-26T04:19:48.998 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:19:48.998 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.998 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:19:48.998 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.998 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:19:48.999 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:19:48.999 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:48.999 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:19:48.999 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:48.999 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:19:48.999 INFO:tasks.workunit.client.0.smithi016.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:19:48.999 INFO:tasks.workunit.client.0.smithi016.stderr:+ sed 's/^/ misc: /' 2024-03-26T04:19:48.999 INFO:tasks.workunit.client.0.smithi016.stderr:+ tee ceph_test_neorados_read_operations.log 2024-03-26T04:19:48.999 INFO:tasks.workunit.client.0.smithi016.stderr:+++++ readlink /proc/75441/exe 2024-03-26T04:19:48.999 INFO:tasks.workunit.client.0.smithi016.stderr:+ tee ceph_test_neorados_misc.log 2024-03-26T04:19:48.999 INFO:tasks.workunit.client.0.smithi016.stderr:++++ basename /usr/bin/bash 2024-03-26T04:19:48.999 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_declare='declare -f' 2024-03-26T04:19:48.999 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_opt=-f 2024-03-26T04:19:48.999 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export which_declare 2024-03-26T04:19:48.999 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export -f which 2024-03-26T04:19:49.000 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset i 2024-03-26T04:19:49.000 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset -f pathmunge 2024-03-26T04:19:49.000 INFO:tasks.workunit.client.0.smithi016.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:19:49.000 INFO:tasks.workunit.client.0.smithi016.stderr:+ export PATH 2024-03-26T04:19:49.000 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:19:49.000 INFO:tasks.workunit.client.0.smithi016.stderr:+ unset rc 2024-03-26T04:19:49.000 INFO:tasks.workunit.client.0.smithi016.stderr:+ tee ceph_test_neorados_pool.log 2024-03-26T04:19:49.000 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:19:49.000 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:19:49.000 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:19:49.000 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:49.000 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:19:49.000 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:49.000 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:19:49.001 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:19:49.001 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:49.001 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:19:49.001 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:49.001 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:19:49.001 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:49.001 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:19:49.001 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:49.001 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:19:49.001 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup 2024-03-26T04:19:49.001 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:19:49.001 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:19:49.001 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:19:49.001 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:49.001 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:19:49.002 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:49.002 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:19:49.002 INFO:tasks.workunit.client.0.smithi016.stderr:+ sed 's/^/ pool: /' 2024-03-26T04:19:49.002 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:19:49.002 INFO:tasks.workunit.client.0.smithi016.stderr:+++ return 2024-03-26T04:19:49.002 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:49.002 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:19:49.002 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:49.002 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:19:49.002 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export S_COLORS=auto 2024-03-26T04:19:49.002 INFO:tasks.workunit.client.0.smithi016.stderr:+++ S_COLORS=auto 2024-03-26T04:19:49.002 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:49.002 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:19:49.002 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:49.003 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:19:49.003 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:49.003 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph_test_neorados_pool 2024-03-26T04:19:49.003 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:19:49.003 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:19:49.003 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:19:49.003 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:49.003 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:19:49.003 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:49.003 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:19:49.003 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:19:49.003 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:49.003 INFO:tasks.workunit.client.0.smithi016.stderr:++ awk '{print $1}' 2024-03-26T04:19:49.003 INFO:tasks.workunit.client.0.smithi016.stderr:++ echo write_operations 2024-03-26T04:19:49.004 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/locale 2024-03-26T04:19:49.004 INFO:tasks.workunit.client.0.smithi016.stdout:test write_operations on pid 75583 2024-03-26T04:19:49.011 INFO:tasks.workunit.client.0.smithi016.stderr:+ ff=write_operations 2024-03-26T04:19:49.012 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=75583 2024-03-26T04:19:49.012 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'test write_operations on pid 75583' 2024-03-26T04:19:49.012 INFO:tasks.workunit.client.0.smithi016.stderr:+ pids[$f]=75583 2024-03-26T04:19:49.012 INFO:tasks.workunit.client.0.smithi016.stderr:+ ret=0 2024-03-26T04:19:49.012 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' 1 -eq 1 ']' 2024-03-26T04:19:49.012 INFO:tasks.workunit.client.0.smithi016.stderr:+ for t in "${!pids[@]}" 2024-03-26T04:19:49.012 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74360 2024-03-26T04:19:49.012 INFO:tasks.workunit.client.0.smithi016.stderr:+ wait 74360 2024-03-26T04:19:49.012 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:49.012 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:19:49.012 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:49.012 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:19:49.012 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:19:49.013 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:19:49.013 INFO:tasks.workunit.client.0.smithi016.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:19:49.013 INFO:tasks.workunit.client.0.smithi016.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:19:49.013 INFO:tasks.workunit.client.0.smithi016.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:19:49.013 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:49.013 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:19:49.013 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:49.013 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:19:49.013 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup config 2024-03-26T04:19:49.013 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:49.013 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:49.013 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:19:49.013 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:49.013 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:19:49.014 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:49.014 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:19:49.014 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:19:49.014 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:49.014 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:19:49.014 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:49.014 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:19:49.014 INFO:tasks.workunit.client.0.smithi016.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:19:49.014 INFO:tasks.workunit.client.0.smithi016.stderr:+++++ readlink /proc/75535/exe 2024-03-26T04:19:49.014 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:19:49.014 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:19:49.014 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:19:49.014 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:49.015 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:19:49.015 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:49.015 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:19:49.015 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:19:49.015 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:49.015 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:19:49.015 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:49.015 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:19:49.015 INFO:tasks.workunit.client.0.smithi016.stderr:++++ basename /usr/bin/bash 2024-03-26T04:19:49.015 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:49.015 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:19:49.015 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:49.015 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:19:49.015 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup 2024-03-26T04:19:49.016 INFO:tasks.workunit.client.0.smithi016.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2024-03-26T04:19:49.016 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_declare='declare -f' 2024-03-26T04:19:49.016 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_opt=-f 2024-03-26T04:19:49.016 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export which_declare 2024-03-26T04:19:49.016 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export -f which 2024-03-26T04:19:49.016 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset i 2024-03-26T04:19:49.016 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset -f pathmunge 2024-03-26T04:19:49.016 INFO:tasks.workunit.client.0.smithi016.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:19:49.016 INFO:tasks.workunit.client.0.smithi016.stderr:+ export PATH 2024-03-26T04:19:49.016 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:19:49.016 INFO:tasks.workunit.client.0.smithi016.stderr:+ unset rc 2024-03-26T04:19:49.016 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -f /etc/bashrc ']' 2024-03-26T04:19:49.016 INFO:tasks.workunit.client.0.smithi016.stderr:+ . /etc/bashrc 2024-03-26T04:19:49.016 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph_test_neorados_watch_notify 2024-03-26T04:19:49.016 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -z '' ']' 2024-03-26T04:19:49.017 INFO:tasks.workunit.client.0.smithi016.stderr:++ BASHRCSOURCED=Y 2024-03-26T04:19:49.017 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:49.017 INFO:tasks.workunit.client.0.smithi016.stderr:++ shopt -q login_shell 2024-03-26T04:19:49.017 INFO:tasks.workunit.client.0.smithi016.stderr:+++ umask 2024-03-26T04:19:49.017 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' 0022 -eq 0 ']' 2024-03-26T04:19:49.017 INFO:tasks.workunit.client.0.smithi016.stderr:++ SHELL=/bin/bash 2024-03-26T04:19:49.017 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:49.017 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-03-26T04:19:49.017 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:49.017 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/bash_completion.sh 2024-03-26T04:19:49.017 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-03-26T04:19:49.017 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:49.017 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-03-26T04:19:49.017 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:49.018 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/cobbler.sh 2024-03-26T04:19:49.018 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:49.018 INFO:tasks.workunit.client.0.smithi016.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-03-26T04:19:49.018 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:49.018 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-03-26T04:19:49.018 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:49.018 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorgrep.sh 2024-03-26T04:19:49.018 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:49.018 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/locale 2024-03-26T04:19:49.018 INFO:tasks.workunit.client.0.smithi016.stderr:+ sed 's/^/ watch_notify: /' 2024-03-26T04:19:49.018 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'grep=grep --color=auto' 2024-03-26T04:19:49.018 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'egrep=egrep --color=auto' 2024-03-26T04:19:49.018 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-03-26T04:19:49.018 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:49.019 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-03-26T04:19:49.019 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:49.019 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorls.sh 2024-03-26T04:19:49.019 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' '!' -t 0 ']' 2024-03-26T04:19:49.019 INFO:tasks.workunit.client.0.smithi016.stderr:+++ return 2024-03-26T04:19:49.019 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:49.019 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-03-26T04:19:49.019 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:49.019 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-03-26T04:19:49.019 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export S_COLORS=auto 2024-03-26T04:19:49.019 INFO:tasks.workunit.client.0.smithi016.stderr:+++ S_COLORS=auto 2024-03-26T04:19:49.019 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:49.019 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-03-26T04:19:49.019 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:49.020 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-03-26T04:19:49.020 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:49.020 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:49.020 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:19:49.020 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:49.020 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:19:49.020 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:19:49.020 INFO:tasks.workunit.client.0.smithi016.stderr:+ tee ceph_test_neorados_watch_notify.log 2024-03-26T04:19:49.020 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:19:49.020 INFO:tasks.workunit.client.0.smithi016.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:19:49.020 INFO:tasks.workunit.client.0.smithi016.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:19:49.020 INFO:tasks.workunit.client.0.smithi016.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:19:49.020 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:49.021 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:19:49.021 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:49.021 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:19:49.021 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup config 2024-03-26T04:19:49.021 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:49.021 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:49.021 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:19:49.021 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:49.021 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:19:49.021 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:49.021 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:19:49.021 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:19:49.021 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:49.021 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:19:49.022 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:49.022 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:19:49.022 INFO:tasks.workunit.client.0.smithi016.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:19:49.022 INFO:tasks.workunit.client.0.smithi016.stderr:+++++ readlink /proc/75521/exe 2024-03-26T04:19:49.022 INFO:tasks.workunit.client.0.smithi016.stderr:++++ basename /usr/bin/bash 2024-03-26T04:19:49.022 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_declare='declare -f' 2024-03-26T04:19:49.022 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_opt=-f 2024-03-26T04:19:49.022 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export which_declare 2024-03-26T04:19:49.022 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export -f which 2024-03-26T04:19:49.022 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset i 2024-03-26T04:19:49.022 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset -f pathmunge 2024-03-26T04:19:49.022 INFO:tasks.workunit.client.0.smithi016.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:19:49.022 INFO:tasks.workunit.client.0.smithi016.stderr:+ export PATH 2024-03-26T04:19:49.022 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:19:49.023 INFO:tasks.workunit.client.0.smithi016.stderr:+ unset rc 2024-03-26T04:19:49.023 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph_test_neorados_snapshots 2024-03-26T04:19:49.023 INFO:tasks.workunit.client.0.smithi016.stderr:+ sed 's/^/ snapshots: /' 2024-03-26T04:19:49.023 INFO:tasks.workunit.client.0.smithi016.stderr:+ tee ceph_test_neorados_snapshots.log 2024-03-26T04:19:49.023 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-03-26T04:19:49.023 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-03-26T04:19:49.023 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-03-26T04:19:49.023 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:49.023 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-03-26T04:19:49.023 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:49.023 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-03-26T04:19:49.023 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-03-26T04:19:49.023 INFO:tasks.workunit.client.0.smithi016.stderr:+++ /usr/libexec/grepconf.sh -c 2024-03-26T04:19:49.023 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-03-26T04:19:49.023 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-03-26T04:19:49.024 INFO:tasks.workunit.client.0.smithi016.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-03-26T04:19:49.024 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:49.024 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-03-26T04:19:49.024 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:49.024 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/debuginfod.sh 2024-03-26T04:19:49.024 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-03-26T04:19:49.024 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:49.024 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-03-26T04:19:49.024 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:49.024 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/gawk.sh 2024-03-26T04:19:49.024 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:49.024 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-03-26T04:19:49.024 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:49.024 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/lang.sh 2024-03-26T04:19:49.025 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup 2024-03-26T04:19:49.025 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/locale 2024-03-26T04:19:49.025 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:49.025 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG_backup=en_US.UTF-8 2024-03-26T04:19:49.025 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:49.025 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /etc/locale.conf ']' 2024-03-26T04:19:49.025 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -x /usr/bin/sed ']' 2024-03-26T04:19:49.025 INFO:tasks.workunit.client.0.smithi016.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-03-26T04:19:49.025 INFO:tasks.workunit.client.0.smithi016.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-03-26T04:19:49.025 INFO:tasks.workunit.client.0.smithi016.stderr:++++ export LANG=en_US.UTF-8 2024-03-26T04:19:49.025 INFO:tasks.workunit.client.0.smithi016.stderr:++++ LANG=en_US.UTF-8 2024-03-26T04:19:49.025 INFO:tasks.workunit.client.0.smithi016.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-03-26T04:19:49.025 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-03-26T04:19:49.025 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:49.026 INFO:tasks.workunit.client.0.smithi016.stderr:+++ LANG=en_US.UTF-8 2024-03-26T04:19:49.026 INFO:tasks.workunit.client.0.smithi016.stderr:+++ unset LANG_backup config 2024-03-26T04:19:49.026 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n '' ']' 2024-03-26T04:19:49.026 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-03-26T04:19:49.026 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' dumb = linux ']' 2024-03-26T04:19:49.026 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:49.026 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-03-26T04:19:49.026 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:49.026 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/less.sh 2024-03-26T04:19:49.026 INFO:tasks.workunit.client.0.smithi016.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-03-26T04:19:49.026 INFO:tasks.workunit.client.0.smithi016.stderr:++ for i in /etc/profile.d/*.sh 2024-03-26T04:19:49.026 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-03-26T04:19:49.026 INFO:tasks.workunit.client.0.smithi016.stderr:++ '[' '' ']' 2024-03-26T04:19:49.027 INFO:tasks.workunit.client.0.smithi016.stderr:++ . /etc/profile.d/which2.sh 2024-03-26T04:19:49.027 INFO:tasks.workunit.client.0.smithi016.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-03-26T04:19:49.027 INFO:tasks.workunit.client.0.smithi016.stderr:+++++ readlink /proc/75583/exe 2024-03-26T04:19:49.027 INFO:tasks.workunit.client.0.smithi016.stderr:++++ basename /usr/bin/bash 2024-03-26T04:19:49.027 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_declare='declare -f' 2024-03-26T04:19:49.027 INFO:tasks.workunit.client.0.smithi016.stderr:+++ which_opt=-f 2024-03-26T04:19:49.027 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export which_declare 2024-03-26T04:19:49.027 INFO:tasks.workunit.client.0.smithi016.stderr:+++ export -f which 2024-03-26T04:19:49.027 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset i 2024-03-26T04:19:49.027 INFO:tasks.workunit.client.0.smithi016.stderr:++ unset -f pathmunge 2024-03-26T04:19:49.027 INFO:tasks.workunit.client.0.smithi016.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-03-26T04:19:49.027 INFO:tasks.workunit.client.0.smithi016.stderr:+ export PATH 2024-03-26T04:19:49.027 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-03-26T04:19:49.027 INFO:tasks.workunit.client.0.smithi016.stderr:+ unset rc 2024-03-26T04:19:49.028 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph_test_neorados_write_operations 2024-03-26T04:19:49.028 INFO:tasks.workunit.client.0.smithi016.stderr:+ tee ceph_test_neorados_write_operations.log 2024-03-26T04:19:49.028 INFO:tasks.workunit.client.0.smithi016.stderr:+ sed 's/^/ write_operations: /' 2024-03-26T04:19:49.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:19:48 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:19:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:19:52.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:19:52 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:19:54.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:53 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3950679362' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi016-74781-1"}]: dispatch 2024-03-26T04:19:54.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:53 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3850201903' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-03-26T04:19:54.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:53 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3950679362' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi016-74781-1"}]: dispatch 2024-03-26T04:19:54.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:53 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3850201903' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-03-26T04:19:54.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:53 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3950679362' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi016-74781-1"}]: dispatch 2024-03-26T04:19:54.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:53 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3850201903' entity='client.admin' cmd=[{"prefix":"quorum_status"}]: dispatch 2024-03-26T04:19:57.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:57 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3264354453' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi016-75488-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:19:57.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:57 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2219155529' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi016-75529-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:19:57.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:57 smithi183 ceph-mon[33745]: from='client.34742 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi016-74781-1"}]: dispatch 2024-03-26T04:19:57.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:57 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2322258890' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi016-74285-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:57.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:57 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:19:57.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:57 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3950679362' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi016-74781-1"}]: dispatch 2024-03-26T04:19:57.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:57 smithi183 ceph-mon[33745]: osdmap e73: 8 total, 8 up, 8 in 2024-03-26T04:19:57.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:57 smithi183 ceph-mon[33745]: from='client.34742 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi016-74781-1"}]: dispatch 2024-03-26T04:19:57.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:57 smithi183 ceph-mon[33745]: from='client.24901 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi016-75488-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:19:57.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:57 smithi183 ceph-mon[33745]: from='client.24904 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi016-75529-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:19:57.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:57 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2993882458' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi016-74264-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:57.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:57 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3264354453' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi016-75488-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:19:57.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:57 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2219155529' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi016-75529-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:19:57.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:57 smithi016 ceph-mon[29101]: from='client.34742 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi016-74781-1"}]: dispatch 2024-03-26T04:19:57.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:57 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2322258890' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi016-74285-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:57.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:57 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:19:57.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:57 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3950679362' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi016-74781-1"}]: dispatch 2024-03-26T04:19:57.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:57 smithi016 ceph-mon[29101]: osdmap e73: 8 total, 8 up, 8 in 2024-03-26T04:19:57.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:57 smithi016 ceph-mon[29101]: from='client.34742 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi016-74781-1"}]: dispatch 2024-03-26T04:19:57.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:57 smithi016 ceph-mon[29101]: from='client.24901 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi016-75488-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:19:57.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:57 smithi016 ceph-mon[29101]: from='client.24904 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi016-75529-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:19:57.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:57 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2993882458' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi016-74264-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:57.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:57 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3264354453' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi016-75488-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:19:57.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:57 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2219155529' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi016-75529-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:19:57.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:57 smithi016 ceph-mon[37215]: from='client.34742 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi016-74781-1"}]: dispatch 2024-03-26T04:19:57.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:57 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2322258890' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi016-74285-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:57.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:57 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:19:57.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:57 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3950679362' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi016-74781-1"}]: dispatch 2024-03-26T04:19:57.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:57 smithi016 ceph-mon[37215]: osdmap e73: 8 total, 8 up, 8 in 2024-03-26T04:19:57.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:57 smithi016 ceph-mon[37215]: from='client.34742 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi016-74781-1"}]: dispatch 2024-03-26T04:19:57.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:57 smithi016 ceph-mon[37215]: from='client.24901 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi016-75488-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:19:57.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:57 smithi016 ceph-mon[37215]: from='client.24904 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi016-75529-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:19:57.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:57 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2993882458' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBig_smithi016-74264-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:58.619 INFO:tasks.workunit.client.0.smithi016.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-03-26T04:19:58.619 INFO:tasks.workunit.client.0.smithi016.stdout: api_asio: [----------] Global test environment set-up. 2024-03-26T04:19:58.620 INFO:tasks.workunit.client.0.smithi016.stdout: api_asio: [----------] 12 tests from AsioRados 2024-03-26T04:19:58.620 INFO:tasks.workunit.client.0.smithi016.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-03-26T04:19:58.620 INFO:tasks.workunit.client.0.smithi016.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (1 ms) 2024-03-26T04:19:58.620 INFO:tasks.workunit.client.0.smithi016.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-03-26T04:19:58.620 INFO:tasks.workunit.client.0.smithi016.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (1 ms) 2024-03-26T04:19:58.620 INFO:tasks.workunit.client.0.smithi016.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-03-26T04:19:58.620 INFO:tasks.workunit.client.0.smithi016.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (1 ms) 2024-03-26T04:19:58.620 INFO:tasks.workunit.client.0.smithi016.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-03-26T04:19:58.620 INFO:tasks.workunit.client.0.smithi016.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (8 ms) 2024-03-26T04:19:58.620 INFO:tasks.workunit.client.0.smithi016.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-03-26T04:19:58.620 INFO:tasks.workunit.client.0.smithi016.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (8 ms) 2024-03-26T04:19:58.620 INFO:tasks.workunit.client.0.smithi016.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-03-26T04:19:58.620 INFO:tasks.workunit.client.0.smithi016.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (7 ms) 2024-03-26T04:19:58.620 INFO:tasks.workunit.client.0.smithi016.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-03-26T04:19:58.621 INFO:tasks.workunit.client.0.smithi016.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (1 ms) 2024-03-26T04:19:58.621 INFO:tasks.workunit.client.0.smithi016.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-03-26T04:19:58.621 INFO:tasks.workunit.client.0.smithi016.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (1 ms) 2024-03-26T04:19:58.621 INFO:tasks.workunit.client.0.smithi016.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-03-26T04:19:58.621 INFO:tasks.workunit.client.0.smithi016.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (1 ms) 2024-03-26T04:19:58.621 INFO:tasks.workunit.client.0.smithi016.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-03-26T04:19:58.621 INFO:tasks.workunit.client.0.smithi016.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (3 ms) 2024-03-26T04:19:58.621 INFO:tasks.workunit.client.0.smithi016.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-03-26T04:19:58.621 INFO:tasks.workunit.client.0.smithi016.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (3 ms) 2024-03-26T04:19:58.621 INFO:tasks.workunit.client.0.smithi016.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-03-26T04:19:58.621 INFO:tasks.workunit.client.0.smithi016.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (4 ms) 2024-03-26T04:19:58.621 INFO:tasks.workunit.client.0.smithi016.stdout: api_asio: [----------] 12 tests from AsioRados (39 ms total) 2024-03-26T04:19:58.621 INFO:tasks.workunit.client.0.smithi016.stdout: api_asio: 2024-03-26T04:19:58.621 INFO:tasks.workunit.client.0.smithi016.stdout: api_asio: [----------] Global test environment tear-down 2024-03-26T04:19:58.621 INFO:tasks.workunit.client.0.smithi016.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (10045 ms total) 2024-03-26T04:19:58.622 INFO:tasks.workunit.client.0.smithi016.stdout: api_asio: [ PASSED ] 12 tests. 2024-03-26T04:19:58.943 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-03-26T04:19:58.943 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: [----------] Global test environment set-up. 2024-03-26T04:19:58.943 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: [----------] 7 tests from LibRadosList 2024-03-26T04:19:58.943 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-03-26T04:19:58.943 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: [ OK ] LibRadosList.ListObjects (395 ms) 2024-03-26T04:19:58.943 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-03-26T04:19:58.943 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (38 ms) 2024-03-26T04:19:58.943 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-03-26T04:19:58.943 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: myset foo1,foo2,foo3 2024-03-26T04:19:58.943 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: foo1 2024-03-26T04:19:58.943 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: foo2 2024-03-26T04:19:58.943 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: foo3 2024-03-26T04:19:58.944 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: myset foo1,foo4,foo5 2024-03-26T04:19:58.944 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: foo4 2024-03-26T04:19:58.944 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: foo5 2024-03-26T04:19:58.944 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: foo1 2024-03-26T04:19:58.944 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: myset foo6,foo7 2024-03-26T04:19:58.944 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: foo7 2024-03-26T04:19:58.944 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: foo6 2024-03-26T04:19:58.944 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-03-26T04:19:58.944 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: ns1:foo4 2024-03-26T04:19:58.944 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: ns1:foo5 2024-03-26T04:19:58.944 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: ns2:foo7 2024-03-26T04:19:58.944 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: ns2:foo6 2024-03-26T04:19:58.944 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: ns1:foo1 2024-03-26T04:19:58.944 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: :foo1 2024-03-26T04:19:58.945 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: :foo2 2024-03-26T04:19:58.945 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: :foo3 2024-03-26T04:19:58.945 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (86 ms) 2024-03-26T04:19:58.945 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-03-26T04:19:58.945 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: 1 0 2024-03-26T04:19:58.945 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: 10 0 2024-03-26T04:19:58.945 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: 13 0 2024-03-26T04:19:58.945 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: 7 0 2024-03-26T04:19:58.945 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: 14 0 2024-03-26T04:19:58.945 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: 0 0 2024-03-26T04:19:58.945 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: 15 0 2024-03-26T04:19:58.945 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: 11 0 2024-03-26T04:19:58.945 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: 5 0 2024-03-26T04:19:58.945 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: 8 0 2024-03-26T04:19:58.946 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: 6 0 2024-03-26T04:19:58.946 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: 3 0 2024-03-26T04:19:58.946 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: 4 0 2024-03-26T04:19:58.946 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: 12 0 2024-03-26T04:19:58.946 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: 9 0 2024-03-26T04:19:58.946 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: 2 0 2024-03-26T04:19:58.946 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-03-26T04:19:58.946 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (68 ms) 2024-03-26T04:19:58.946 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-03-26T04:19:58.946 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: x cursor=MIN 2024-03-26T04:19:58.946 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > oid=1 cursor=16:02547ec2:::1:head 2024-03-26T04:19:58.946 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > oid=10 cursor=16:52ea6a34:::10:head 2024-03-26T04:19:58.946 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > oid=13 cursor=16:566253c9:::13:head 2024-03-26T04:19:58.947 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > oid=7 cursor=16:5c6b0b28:::7:head 2024-03-26T04:19:58.947 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > oid=14 cursor=16:62a1935d:::14:head 2024-03-26T04:19:58.947 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > oid=0 cursor=16:6cac518f:::0:head 2024-03-26T04:19:58.947 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > oid=15 cursor=16:863748b0:::15:head 2024-03-26T04:19:58.947 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > oid=11 cursor=16:89d3ae78:::11:head 2024-03-26T04:19:58.947 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > oid=5 cursor=16:b29083e3:::5:head 2024-03-26T04:19:58.947 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > oid=8 cursor=16:bd63b0f1:::8:head 2024-03-26T04:19:58.947 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > oid=6 cursor=16:c4fdafeb:::6:head 2024-03-26T04:19:58.947 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > oid=3 cursor=16:cfc208b3:::3:head 2024-03-26T04:19:58.947 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > oid=4 cursor=16:d83876eb:::4:head 2024-03-26T04:19:58.947 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > oid=12 cursor=16:de5d7c5f:::12:head 2024-03-26T04:19:58.947 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > oid=9 cursor=16:e960b815:::9:head 2024-03-26T04:19:58.947 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > oid=2 cursor=16:f905c69b:::2:head 2024-03-26T04:19:58.947 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: FIRST> seek to MIN oid=1 2024-03-26T04:19:58.947 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : oid=1 cursor=16:02547ec2:::1:head 2024-03-26T04:19:58.947 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : seek to 16:02547ec2:::1:head 2024-03-26T04:19:58.947 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > 16:02547ec2:::1:head -> 1 2024-03-26T04:19:58.948 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : oid=10 cursor=16:52ea6a34:::10:head 2024-03-26T04:19:58.948 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : seek to 16:52ea6a34:::10:head 2024-03-26T04:19:58.948 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > 16:52ea6a34:::10:head -> 10 2024-03-26T04:19:58.948 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : oid=13 cursor=16:566253c9:::13:head 2024-03-26T04:19:58.948 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : seek to 16:566253c9:::13:head 2024-03-26T04:19:58.948 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > 16:566253c9:::13:head -> 13 2024-03-26T04:19:58.948 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : oid=7 cursor=16:5c6b0b28:::7:head 2024-03-26T04:19:58.948 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : seek to 16:5c6b0b28:::7:head 2024-03-26T04:19:58.948 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > 16:5c6b0b28:::7:head -> 7 2024-03-26T04:19:59.009 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : oid=14 cursor=16:62a1935d:::14:head 2024-03-26T04:19:59.009 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : seek to 16:62a1935d:::14:head 2024-03-26T04:19:59.009 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > 16:62a1935d:::14:head -> 14 2024-03-26T04:19:59.009 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : oid=0 cursor=16:6cac518f:::0:head 2024-03-26T04:19:59.009 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : seek to 16:6cac518f:::0:head 2024-03-26T04:19:59.010 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > 16:6cac518f:::0:head -> 0 2024-03-26T04:19:59.010 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : oid=15 cursor=16:863748b0:::15:head 2024-03-26T04:19:59.010 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : seek to 16:863748b0:::15:head 2024-03-26T04:19:59.010 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > 16:863748b0:::15:head -> 15 2024-03-26T04:19:59.010 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : oid=11 cursor=16:89d3ae78:::11:head 2024-03-26T04:19:59.010 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : seek to 16:89d3ae78:::11:head 2024-03-26T04:19:59.010 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > 16:89d3ae78:::11:head -> 11 2024-03-26T04:19:59.010 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : oid=5 cursor=16:b29083e3:::5:head 2024-03-26T04:19:59.010 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : seek to 16:b29083e3:::5:head 2024-03-26T04:19:59.010 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > 16:b29083e3:::5:head -> 5 2024-03-26T04:19:59.010 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : oid=8 cursor=16:bd63b0f1:::8:head 2024-03-26T04:19:59.010 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : seek to 16:bd63b0f1:::8:head 2024-03-26T04:19:59.010 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > 16:bd63b0f1:::8:head -> 8 2024-03-26T04:19:59.011 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : oid=6 cursor=16:c4fdafeb:::6:head 2024-03-26T04:19:59.011 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : seek to 16:c4fdafeb:::6:head 2024-03-26T04:19:59.011 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > 16:c4fdafeb:::6:head -> 6 2024-03-26T04:19:59.011 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : oid=3 cursor=16:cfc208b3:::3:head 2024-03-26T04:19:59.011 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : seek to 16:cfc208b3:::3:head 2024-03-26T04:19:59.011 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > 16:cfc208b3:::3:head -> 3 2024-03-26T04:19:59.011 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : oid=4 cursor=16:d83876eb:::4:head 2024-03-26T04:19:59.011 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : seek to 16:d83876eb:::4:head 2024-03-26T04:19:59.011 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > 16:d83876eb:::4:head -> 4 2024-03-26T04:19:59.011 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : oid=12 cursor=16:de5d7c5f:::12:head 2024-03-26T04:19:59.011 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : seek to 16:de5d7c5f:::12:head 2024-03-26T04:19:59.011 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > 16:de5d7c5f:::12:head -> 12 2024-03-26T04:19:59.011 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : oid=9 cursor=16:e960b815:::9:head 2024-03-26T04:19:59.011 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : seek to 16:e960b815:::9:head 2024-03-26T04:19:59.012 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > 16:e960b815:::9:head -> 9 2024-03-26T04:19:59.012 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : oid=2 cursor=16:f905c69b:::2:head 2024-03-26T04:19:59.012 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : seek to 16:f905c69b:::2:head 2024-03-26T04:19:59.012 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > 16:f905c69b:::2:head -> 2 2024-03-26T04:19:59.012 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : seek to 16:d83876eb:::4:head 2024-03-26T04:19:59.012 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : cursor()=16:d83876eb:::4:head expected=16:d83876eb:::4:head 2024-03-26T04:19:59.012 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > 16:d83876eb:::4:head -> 4 2024-03-26T04:19:59.012 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : entry=4 expected=4 2024-03-26T04:19:59.012 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : seek to 16:cfc208b3:::3:head 2024-03-26T04:19:59.012 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : cursor()=16:cfc208b3:::3:head expected=16:cfc208b3:::3:head 2024-03-26T04:19:59.012 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > 16:cfc208b3:::3:head -> 3 2024-03-26T04:19:59.012 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : entry=3 expected=3 2024-03-26T04:19:59.012 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : seek to 16:c4fdafeb:::6:head 2024-03-26T04:19:59.012 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : cursor()=16:c4fdafeb:::6:head expected=16:c4fdafeb:::6:head 2024-03-26T04:19:59.013 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > 16:c4fdafeb:::6:head -> 6 2024-03-26T04:19:59.013 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : entry=6 expected=6 2024-03-26T04:19:59.013 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : seek to 16:863748b0:::15:head 2024-03-26T04:19:59.013 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : cursor()=16:863748b0:::15:head expected=16:863748b0:::15:head 2024-03-26T04:19:59.013 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > 16:863748b0:::15:head -> 15 2024-03-26T04:19:59.013 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : entry=15 expected=15 2024-03-26T04:19:59.013 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : seek to 16:6cac518f:::0:head 2024-03-26T04:19:59.013 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : cursor()=16:6cac518f:::0:head expected=16:6cac518f:::0:head 2024-03-26T04:19:59.013 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > 16:6cac518f:::0:head -> 0 2024-03-26T04:19:59.013 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : entry=0 expected=0 2024-03-26T04:19:59.013 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : seek to 16:62a1935d:::14:head 2024-03-26T04:19:59.013 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : cursor()=16:62a1935d:::14:head expected=16:62a1935d:::14:head 2024-03-26T04:19:59.013 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > 16:62a1935d:::14:head -> 14 2024-03-26T04:19:59.013 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : entry=14 expected=14 2024-03-26T04:19:59.013 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : seek to 16:566253c9:::13:head 2024-03-26T04:19:59.014 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : cursor()=16:566253c9:::13:head expected=16:566253c9:::13:head 2024-03-26T04:19:59.014 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > 16:566253c9:::13:head -> 13 2024-03-26T04:19:59.014 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : entry=13 expected=13 2024-03-26T04:19:59.014 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : seek to 16:f905c69b:::2:head 2024-03-26T04:19:59.014 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : cursor()=16:f905c69b:::2:head expected=16:f905c69b:::2:head 2024-03-26T04:19:59.014 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > 16:f905c69b:::2:head -> 2 2024-03-26T04:19:59.136 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : e list_parallel: process_1_[75078]: starting. 2024-03-26T04:19:59.136 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_1_[75078]: creating pool ceph_test_rados_list_parallel.smithi016-75045 2024-03-26T04:19:59.136 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_1_[75078]: created object 0... 2024-03-26T04:19:59.136 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_1_[75078]: created object 25... 2024-03-26T04:19:59.136 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_1_[75078]: created object 49... 2024-03-26T04:19:59.137 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_1_[75078]: finishing. 2024-03-26T04:19:59.137 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_1_[75078]: shutting down. 2024-03-26T04:19:59.137 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_2_[75080]: starting. 2024-03-26T04:19:59.137 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_2_[75080]: listing objects. 2024-03-26T04:19:59.137 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_2_[75080]: listed object 0... 2024-03-26T04:19:59.137 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_2_[75080]: listed object 25... 2024-03-26T04:19:59.137 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_2_[75080]: saw 50 objects 2024-03-26T04:19:59.137 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_2_[75080]: shutting down. 2024-03-26T04:19:59.137 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: ******************************* 2024-03-26T04:19:59.137 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_3_[76250]: starting. 2024-03-26T04:19:59.137 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_3_[76250]: creating pool ceph_test_rados_list_parallel.smithi016-75045 2024-03-26T04:19:59.137 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_3_[76250]: created object 0... 2024-03-26T04:19:59.137 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_3_[76250]: created object 25... 2024-03-26T04:19:59.137 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_3_[76250]: created object 49... 2024-03-26T04:19:59.138 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_3_[76250]: finishing. 2024-03-26T04:19:59.138 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_3_[76250]: shutting down. 2024-03-26T04:19:59.138 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: ******************************* 2024-03-26T04:19:59.138 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_4_[76251]: starting. 2024-03-26T04:19:59.138 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_4_[76251]: listing objects. 2024-03-26T04:19:59.138 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_4_[76251]: listed object 0... 2024-03-26T04:19:59.138 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_4_[76251]: listed object 25... 2024-03-26T04:19:59.138 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_4_[76251]: saw 43 objects 2024-03-26T04:19:59.138 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_4_[76251]: shutting down. 2024-03-26T04:19:59.138 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: ******************************* 2024-03-26T04:19:59.138 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_5_[76252]: starting. 2024-03-26T04:19:59.138 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_5_[76252]: removed 25 objects... 2024-03-26T04:19:59.138 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_5_[76252]: removed half of the objects 2024-03-26T04:19:59.138 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_5_[76252]: removed 50 objects... 2024-03-26T04:19:59.138 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_5_[76252]: removed 50 objects 2024-03-26T04:19:59.139 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_5_[76252]: shutting down. 2024-03-26T04:19:59.139 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: ******************************* 2024-03-26T04:19:59.139 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: ******************************* 2024-03-26T04:19:59.139 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_6_[76299]: starting. 2024-03-26T04:19:59.139 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_6_[76299]: creating pool ceph_test_rados_list_parallel.smithi016-75045 2024-03-26T04:19:59.139 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_6_[76299]: created object 0... 2024-03-26T04:19:59.139 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_6_[76299]: created object 25... 2024-03-26T04:19:59.139 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_6_[76299]: created object 49... 2024-03-26T04:19:59.139 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_6_[76299]: finishing. 2024-03-26T04:19:59.139 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_6_[76299]: shutting down. 2024-03-26T04:19:59.139 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: ******************************* 2024-03-26T04:19:59.139 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: ******************************* 2024-03-26T04:19:59.139 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_7_[76300]: starting. 2024-03-26T04:19:59.139 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_7_[76300]: listing objects. 2024-03-26T04:19:59.139 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_7_[76300]: listed object 0... 2024-03-26T04:19:59.140 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_7_[76300]: listed object 25... 2024-03-26T04:19:59.140 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_7_[76300]: listed object 50... 2024-03-26T04:19:59.140 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_7_[76300]: saw 56 objects 2024-03-26T04:19:59.140 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_7_[76300]: shutting down. 2024-03-26T04:19:59.140 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: ******************************* 2024-03-26T04:19:59.140 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: ******************************* 2024-03-26T04:19:59.140 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_8_[76301]: starting. 2024-03-26T04:19:59.140 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_8_[76301]: added 25 objects... 2024-03-26T04:19:59.140 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_8_[76301]: added half of the objects 2024-03-26T04:19:59.140 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_8_[76301]: added 50 objects... 2024-03-26T04:19:59.140 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_8_[76301]: added 50 objects 2024-03-26T04:19:59.140 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_8_[76301]: shutting down. 2024-03-26T04:19:59.140 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: ******************************* 2024-03-26T04:19:59.141 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: ******************************* 2024-03-26T04:19:59.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:19:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:19:59.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:19:59.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: pgmap v66: 196 pgs: 64 unknown, 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:19:59.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:19:59.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: pgmap v67: 196 pgs: 31 creating+peering, 132 active+clean, 33 unknown; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:19:59.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/4099742997' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-75032-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2492186405' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi016-74578-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/4072024826' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi016-74609-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/333457894' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi016-74841-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3738020986' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi016-74362-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1182818240' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi016-74367-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1153924336' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi016-74295-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi016-74374-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/855685679' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi016-74692-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1120834936' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi016-74313-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/622056058' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi016-74514-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/796358913' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi016-74670-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:19:59.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: pgmap v66: 196 pgs: 64 unknown, 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:19:59.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:19:59.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: pgmap v67: 196 pgs: 31 creating+peering, 132 active+clean, 33 unknown; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:19:59.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/4099742997' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-75032-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2492186405' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi016-74578-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/4072024826' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi016-74609-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/333457894' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi016-74841-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.165 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3738020986' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi016-74362-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.165 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1182818240' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi016-74367-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.165 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1153924336' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi016-74295-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.165 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi016-74374-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.165 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/855685679' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi016-74692-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.165 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1120834936' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi016-74313-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.165 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/622056058' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.165 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi016-74514-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.165 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/796358913' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi016-74670-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.165 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2853990369' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi016-74534-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.165 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2516364974' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-26T04:19:59.166 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: pgmap v69: 772 pgs: 31 creating+peering, 132 active+clean, 609 unknown; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:19:59.166 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3950679362' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi016-74781-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:19:59.166 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3068526560' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-26T04:19:59.166 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: osdmap e74: 8 total, 8 up, 8 in 2024-03-26T04:19:59.166 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: pgmap v70: 772 pgs: 342 creating+peering, 132 active+clean, 298 unknown; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:19:59.166 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.34802 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-75032-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.166 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.34700 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi016-74578-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.166 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.34733 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi016-74609-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.166 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.34751 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi016-74841-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.166 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.24673 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi016-74285-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.166 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.34685 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi016-74362-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.166 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.34688 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi016-74367-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.166 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.34742 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi016-74781-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:19:59.167 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.24901 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi016-75488-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:19:59.167 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.24904 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi016-75529-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:19:59.167 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2993882458' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi016-74264-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:19:59.167 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2853990369' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi016-74534-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.168 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2516364974' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-26T04:19:59.168 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: pgmap v69: 772 pgs: 31 creating+peering, 132 active+clean, 609 unknown; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:19:59.168 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3950679362' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi016-74781-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:19:59.168 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3068526560' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-26T04:19:59.168 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: osdmap e74: 8 total, 8 up, 8 in 2024-03-26T04:19:59.169 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: pgmap v70: 772 pgs: 342 creating+peering, 132 active+clean, 298 unknown; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:19:59.169 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.34802 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-75032-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.169 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.34700 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi016-74578-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.169 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.34733 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi016-74609-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.169 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.34751 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi016-74841-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.169 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.24673 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi016-74285-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.169 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.34685 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi016-74362-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.169 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.34688 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi016-74367-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.169 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.34742 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi016-74781-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:19:59.169 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.24901 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi016-75488-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:19:59.170 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.24904 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi016-75529-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:19:59.170 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2993882458' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi016-74264-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:19:59.170 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3264354453' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi016-75488-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi016-75488-1"}]: dispatch 2024-03-26T04:19:59.170 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2219155529' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi016-75529-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi016-75529-1"}]: dispatch 2024-03-26T04:19:59.171 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1067414090' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi016-74987-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.171 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/468960212' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74884-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.171 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3375433492' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74872-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.171 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/927867245' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.171 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: osdmap e75: 8 total, 8 up, 8 in 2024-03-26T04:19:59.171 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3199705519' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-26T04:19:59.171 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.24685 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi016-74295-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.171 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.24709 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi016-74374-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.171 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.24754 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi016-74692-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.171 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.24691 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi016-74313-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.171 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.24730 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.171 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.24733 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi016-74514-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.171 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.24748 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi016-74670-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.171 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.24739 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi016-74534-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.172 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.24901 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi016-75488-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi016-75488-1"}]: dispatch 2024-03-26T04:19:59.172 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.24904 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi016-75529-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi016-75529-1"}]: dispatch 2024-03-26T04:19:59.172 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.24808 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi016-74987-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.172 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.24850 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74884-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.172 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.24859 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74872-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.172 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: from='client.24793 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.172 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[29101]: Health check failed: 20 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:19:59.172 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3264354453' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi016-75488-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi016-75488-1"}]: dispatch 2024-03-26T04:19:59.172 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2219155529' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi016-75529-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi016-75529-1"}]: dispatch 2024-03-26T04:19:59.172 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1067414090' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi016-74987-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.173 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/468960212' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74884-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.173 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3375433492' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74872-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.173 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/927867245' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.173 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: osdmap e75: 8 total, 8 up, 8 in 2024-03-26T04:19:59.173 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3199705519' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-26T04:19:59.173 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.24685 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi016-74295-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.173 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.24709 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi016-74374-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.173 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.24754 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi016-74692-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.173 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.24691 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi016-74313-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.173 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.24730 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.173 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.24733 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi016-74514-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.173 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.24748 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi016-74670-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.173 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.24739 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi016-74534-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.173 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.24901 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi016-75488-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi016-75488-1"}]: dispatch 2024-03-26T04:19:59.174 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.24904 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi016-75529-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi016-75529-1"}]: dispatch 2024-03-26T04:19:59.174 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.24808 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi016-74987-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.174 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.24850 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74884-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.174 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.24859 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74872-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.174 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: from='client.24793 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.174 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:19:58 smithi016 ceph-mon[37215]: Health check failed: 20 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:19:59.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: pgmap v64: 132 pgs: 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:19:59.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: pgmap v66: 196 pgs: 64 unknown, 132 active+clean; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:19:59.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:19:59.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: pgmap v67: 196 pgs: 31 creating+peering, 132 active+clean, 33 unknown; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:19:59.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/4099742997' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-75032-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2492186405' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi016-74578-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/4072024826' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi016-74609-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/333457894' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi016-74841-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3738020986' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi016-74362-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1182818240' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi016-74367-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1153924336' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi016-74295-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi016-74374-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/855685679' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi016-74692-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1120834936' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi016-74313-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/622056058' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi016-74514-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/796358913' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi016-74670-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2853990369' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi016-74534-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2516364974' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-26T04:19:59.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: pgmap v69: 772 pgs: 31 creating+peering, 132 active+clean, 609 unknown; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:19:59.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3950679362' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi016-74781-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:19:59.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3068526560' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-26T04:19:59.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: osdmap e74: 8 total, 8 up, 8 in 2024-03-26T04:19:59.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: pgmap v70: 772 pgs: 342 creating+peering, 132 active+clean, 298 unknown; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:19:59.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.34802 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-75032-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.34700 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi016-74578-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.34733 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi016-74609-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.34751 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi016-74841-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.24673 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi016-74285-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.34685 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi016-74362-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.34688 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi016-74367-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.34742 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi016-74781-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:19:59.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.24901 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritesmithi016-75488-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:19:59.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.24904 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectssmithi016-75529-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:19:59.295 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2993882458' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBig_smithi016-74264-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:19:59.295 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3264354453' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi016-75488-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi016-75488-1"}]: dispatch 2024-03-26T04:19:59.295 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2219155529' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi016-75529-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi016-75529-1"}]: dispatch 2024-03-26T04:19:59.295 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1067414090' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi016-74987-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.295 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/468960212' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74884-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.295 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3375433492' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74872-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.295 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/927867245' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.295 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: osdmap e75: 8 total, 8 up, 8 in 2024-03-26T04:19:59.295 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3199705519' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-26T04:19:59.295 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.24685 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi016-74295-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.295 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.24709 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi016-74374-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.295 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.24754 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi016-74692-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.295 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.24691 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi016-74313-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.295 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.24730 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.296 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.24733 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi016-74514-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.296 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.24748 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi016-74670-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.296 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.24739 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi016-74534-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.296 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.24901 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritesmithi016-75488-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi016-75488-1"}]: dispatch 2024-03-26T04:19:59.296 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.24904 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectssmithi016-75529-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi016-75529-1"}]: dispatch 2024-03-26T04:19:59.296 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.24808 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi016-74987-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.296 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.24850 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74884-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.296 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.24859 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74872-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.296 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: from='client.24793 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:19:59.296 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:19:58 smithi183 ceph-mon[33745]: Health check failed: 20 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:19:59.322 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: ******************************* 2024-03-26T04:19:59.322 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_9_[76376]: starting. 2024-03-26T04:19:59.322 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_9_[76376]: creating pool ceph_test_rados_list_parallel.smithi016-75045 2024-03-26T04:19:59.322 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_9_[76376]: created object 0... 2024-03-26T04:19:59.322 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_9_[76376]: created object 25... 2024-03-26T04:19:59.322 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_9_[76376]: created object 49... 2024-03-26T04:19:59.322 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_9_[76376]: finishing. 2024-03-26T04:19:59.323 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_9_[76376]: shutting down. 2024-03-26T04:19:59.323 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: ******************************* 2024-03-26T04:19:59.323 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: ******************************* 2024-03-26T04:19:59.323 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: ******************************* 2024-03-26T04:19:59.323 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_10_[76377]: starting. 2024-03-26T04:19:59.323 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_10_[76377]: listing objects. 2024-03-26T04:19:59.323 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_10_[76377]: listed object 0... 2024-03-26T04:19:59.323 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_10_[76377]: listed object 25... 2024-03-26T04:19:59.323 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_10_[76377]: listed object 50... 2024-03-26T04:19:59.323 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_10_[76377]: listed object 75... 2024-03-26T04:19:59.323 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_10_[76377]: saw 97 objects 2024-03-26T04:19:59.323 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_10_[76377]: shutting down. 2024-03-26T04:19:59.323 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: ******************************* 2024-03-26T04:19:59.323 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: ******************************* 2024-03-26T04:19:59.324 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: ******************************* 2024-03-26T04:19:59.324 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_13_[76380]: starting. 2024-03-26T04:19:59.324 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_13_[76380]: removed 25 objects... 2024-03-26T04:19:59.324 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_13_[76380]: removed half of the objects 2024-03-26T04:19:59.324 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_13_[76380]: removed 50 objects... 2024-03-26T04:19:59.324 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_13_[76380]: removed 50 objects 2024-03-26T04:19:59.324 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_13_[76380]: shutting down. 2024-03-26T04:19:59.324 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: ******************************* 2024-03-26T04:19:59.324 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: ******************************* 2024-03-26T04:19:59.324 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: ******************************* 2024-03-26T04:19:59.324 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_11_[76378]: starting. 2024-03-26T04:19:59.324 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_11_[76378]: added 25 objects... 2024-03-26T04:19:59.324 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_11_[76378]: added half of the objects 2024-03-26T04:19:59.325 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_11_[76378]: added 50 objects... 2024-03-26T04:19:59.325 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_11_[76378]: added 50 objects 2024-03-26T04:19:59.325 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_11_[76378]: shutting down. 2024-03-26T04:19:59.325 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: ******************************* 2024-03-26T04:19:59.325 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: ******************************* 2024-03-26T04:19:59.325 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: ******************************* 2024-03-26T04:19:59.325 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_12_[76379]: starting. 2024-03-26T04:19:59.325 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_12_[76379]: added 25 objects... 2024-03-26T04:19:59.325 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_12_[76379]: added half of the objects 2024-03-26T04:19:59.325 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_12_[76379]: added 50 objects... 2024-03-26T04:19:59.325 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_12_[76379]: added 50 objects 2024-03-26T04:19:59.325 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_12_[76379]: shutting down. 2024-03-26T04:19:59.325 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: ******************************* 2024-03-26T04:19:59.325 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: ******************************* 2024-03-26T04:19:59.325 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: ******************************* 2024-03-26T04:19:59.326 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: ******************************* 2024-03-26T04:19:59.326 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_14_[76458]: starting. 2024-03-26T04:19:59.326 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_14_[76458]: creating pool ceph_test_rados_list_parallel.smithi016-75045 2024-03-26T04:19:59.326 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_14_[76458]: created object 0... 2024-03-26T04:19:59.326 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_14_[76458]: created object 25... 2024-03-26T04:19:59.326 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_14_[76458]: created object 49... 2024-03-26T04:19:59.326 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_14_[76458]: finishing. 2024-03-26T04:19:59.326 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_14_[76458]: shutting down. 2024-03-26T04:19:59.326 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: ******************************* 2024-03-26T04:19:59.326 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: ******************************* 2024-03-26T04:19:59.326 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: ******************************* 2024-03-26T04:19:59.326 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: ******************************* 2024-03-26T04:19:59.326 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_15_[76459]: starting. 2024-03-26T04:19:59.326 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_15_[76459]: listing objects. 2024-03-26T04:19:59.327 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_15_[76459]: listed object 0... 2024-03-26T04:19:59.764 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_15_[76459]: listed object 2 api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2024-03-26T04:19:59.765 INFO:tasks.workunit.client.0.smithi016.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2024-03-26T04:19:59.765 INFO:tasks.workunit.client.0.smithi016.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2024-03-26T04:19:59.765 INFO:tasks.workunit.client.0.smithi016.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2024-03-26T04:19:59.765 INFO:tasks.workunit.client.0.smithi016.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (10955 ms) 2024-03-26T04:19:59.765 INFO:tasks.workunit.client.0.smithi016.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (10955 ms total) 2024-03-26T04:19:59.765 INFO:tasks.workunit.client.0.smithi016.stdout: api_cls_remote_reads: 2024-03-26T04:19:59.765 INFO:tasks.workunit.client.0.smithi016.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2024-03-26T04:19:59.765 INFO:tasks.workunit.client.0.smithi016.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (10956 ms total) 2024-03-26T04:19:59.765 INFO:tasks.workunit.client.0.smithi016.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2024-03-26T04:20:00.005 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: [==========] Running 4 tests from 1 test suite. 2024-03-26T04:20:00.006 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: [----------] Global test environment set-up. 2024-03-26T04:20:00.006 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: [----------] 4 tests from LibRadosService 2024-03-26T04:20:00.006 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-03-26T04:20:00.006 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5038 ms) 2024-03-26T04:20:00.006 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-03-26T04:20:00.006 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: [ OK ] LibRadosService.RegisterLate (34 ms) 2024-03-26T04:20:00.006 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-03-26T04:20:00.006 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: cluster: 2024-03-26T04:20:00.006 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: id: d7d59826-eb26-11ee-95d0-87774f69a715 2024-03-26T04:20:00.006 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: health: HEALTH_OK 2024-03-26T04:20:00.006 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: 2024-03-26T04:20:00.006 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: services: 2024-03-26T04:20:00.007 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-03-26T04:20:00.007 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-03-26T04:20:00.007 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: osd: 8 osds: 8 up (since 2m), 8 in (since 3m) 2024-03-26T04:20:00.007 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-03-26T04:20:00.007 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: 2024-03-26T04:20:00.007 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: data: 2024-03-26T04:20:00.007 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: pools: 6 pools, 132 pgs 2024-03-26T04:20:00.007 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: objects: 230 objects, 582 KiB 2024-03-26T04:20:00.007 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: usage: 243 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:20:00.007 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: pgs: 132 active+clean 2024-03-26T04:20:00.007 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: 2024-03-26T04:20:00.007 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: io: 2024-03-26T04:20:00.007 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: client: 1.2 KiB/s rd, 1 op/s rd, 0 op/s wr 2024-03-26T04:20:00.007 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: 2024-03-26T04:20:00.008 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: 2024-03-26T04:20:00.008 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: cluster: 2024-03-26T04:20:00.008 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: id: d7d59826-eb26-11ee-95d0-87774f69a715 2024-03-26T04:20:00.008 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: health: HEALTH_OK 2024-03-26T04:20:00.008 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: 2024-03-26T04:20:00.008 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: services: 2024-03-26T04:20:00.008 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-03-26T04:20:00.008 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-03-26T04:20:00.008 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: osd: 8 osds: 8 up (since 2m), 8 in (since 3m) 2024-03-26T04:20:00.008 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-03-26T04:20:00.008 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: 2024-03-26T04:20:00.008 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: data: 2024-03-26T04:20:00.008 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: pools: 6 pools, 132 pgs 2024-03-26T04:20:00.008 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: objects: 230 objects, 582 KiB 2024-03-26T04:20:00.009 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: usage: 243 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:20:00.009 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: pgs: 132 active+clean 2024-03-26T04:20:00.009 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: 2024-03-26T04:20:00.009 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: io: 2024-03-26T04:20:00.009 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: client: 1.2 KiB/s rd, 1 op/s rd, 0 op/s wr 2024-03-26T04:20:00.009 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: 2024-03-26T04:20:00.009 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: 2024-03-26T04:20:00.009 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: cluster: 2024-03-26T04:20:00.009 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: id: d7d59826-eb26-11ee-95d0-87774f69a715 2024-03-26T04:20:00.009 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: health: HEALTH_OK 2024-03-26T04:20:00.009 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: 2024-03-26T04:20:00.009 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: services: 2024-03-26T04:20:00.009 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 6m) 2024-03-26T04:20:00.009 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-03-26T04:20:00.010 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-03-26T04:20:00.010 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-03-26T04:20:00.010 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-03-26T04:20:00.010 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: 2024-03-26T04:20:00.010 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: data: 2024-03-26T04:20:00.010 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: pools: 6 pools, 132 pgs 2024-03-26T04:20:00.010 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: objects: 230 objects, 582 KiB 2024-03-26T04:20:00.010 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: usage: 243 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:20:00.010 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: pgs: 132 active+clean 2024-03-26T04:20:00.010 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: 2024-03-26T04:20:00.010 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: io: 2024-03-26T04:20:00.010 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: client: 853 B/s rd, 0 op/s rd, 0 op/s wr 2024-03-26T04:20:00.011 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: 2024-03-26T04:20:00.011 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: 2024-03-26T04:20:00.011 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: cluster: 2024-03-26T04:20:00.011 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: id: d7d59826-eb26-11ee-95d0-87774f69a715 2024-03-26T04:20:00.011 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: health: HEALTH_WARN 2024-03-26T04:20:00.011 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: 20 pool(s) do not have an application enabled 2024-03-26T04:20:00.011 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: 2024-03-26T04:20:00.011 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: services: 2024-03-26T04:20:00.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[29101]: from='client.34802 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-75032-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[29101]: from='client.34700 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi016-74578-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[29101]: from='client.34733 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi016-74609-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[29101]: from='client.34751 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi016-74841-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[29101]: from='client.24673 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi016-74285-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[29101]: from='client.34685 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi016-74362-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[29101]: from='client.34688 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi016-74367-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[29101]: from='client.34742 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi016-74781-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:20:00.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[29101]: from='client.24685 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi016-74295-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[29101]: from='client.24709 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi016-74374-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[29101]: from='client.24754 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi016-74692-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[29101]: from='client.24691 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi016-74313-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[29101]: from='client.24730 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[29101]: from='client.24733 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi016-74514-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[29101]: from='client.24748 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi016-74670-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[29101]: from='client.24739 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi016-74534-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[29101]: from='client.24808 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi016-74987-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[37215]: from='client.34802 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-75032-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[37215]: from='client.34700 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi016-74578-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[37215]: from='client.34733 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi016-74609-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[37215]: from='client.34751 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi016-74841-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[37215]: from='client.24673 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi016-74285-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[37215]: from='client.34685 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi016-74362-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[37215]: from='client.34688 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi016-74367-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[37215]: from='client.34742 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi016-74781-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:20:00.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[37215]: from='client.24685 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi016-74295-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[37215]: from='client.24709 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi016-74374-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[37215]: from='client.24754 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi016-74692-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[37215]: from='client.24691 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi016-74313-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[37215]: from='client.24730 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[37215]: from='client.24733 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi016-74514-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[37215]: from='client.24748 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi016-74670-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[37215]: from='client.24739 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi016-74534-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[37215]: from='client.24808 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi016-74987-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[29101]: from='client.24850 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74884-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[29101]: from='client.24859 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74872-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[29101]: from='client.24793 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3950679362' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi016-74781-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi016-74781-1"}]: dispatch 2024-03-26T04:20:00.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[29101]: osdmap e76: 8 total, 8 up, 8 in 2024-03-26T04:20:00.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[29101]: from='client.34742 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi016-74781-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi016-74781-1"}]: dispatch 2024-03-26T04:20:00.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:00.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[29101]: from='client.24901 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi016-75488-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi016-75488-1"}]': finished 2024-03-26T04:20:00.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[29101]: from='client.24904 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi016-75529-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi016-75529-1"}]': finished 2024-03-26T04:20:00.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[29101]: osdmap e77: 8 total, 8 up, 8 in 2024-03-26T04:20:00.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3343786834' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi016-74264-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:00.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[29101]: from='client.35075 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi016-74264-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:00.417 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[37215]: from='client.24850 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74884-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.417 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[37215]: from='client.24859 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74872-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.417 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[37215]: from='client.24793 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.417 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3950679362' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi016-74781-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi016-74781-1"}]: dispatch 2024-03-26T04:20:00.417 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[37215]: osdmap e76: 8 total, 8 up, 8 in 2024-03-26T04:20:00.417 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[37215]: from='client.34742 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi016-74781-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi016-74781-1"}]: dispatch 2024-03-26T04:20:00.417 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:00.417 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[37215]: from='client.24901 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi016-75488-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi016-75488-1"}]': finished 2024-03-26T04:20:00.417 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[37215]: from='client.24904 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi016-75529-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi016-75529-1"}]': finished 2024-03-26T04:20:00.417 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[37215]: osdmap e77: 8 total, 8 up, 8 in 2024-03-26T04:20:00.417 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3343786834' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi016-74264-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:00.417 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:00 smithi016 ceph-mon[37215]: from='client.35075 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi016-74264-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:00 smithi183 ceph-mon[33745]: from='client.34802 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-75032-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:00 smithi183 ceph-mon[33745]: from='client.34700 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi016-74578-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:00 smithi183 ceph-mon[33745]: from='client.34733 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi016-74609-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:00 smithi183 ceph-mon[33745]: from='client.34751 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi016-74841-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:00 smithi183 ceph-mon[33745]: from='client.24673 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi016-74285-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:00 smithi183 ceph-mon[33745]: from='client.34685 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi016-74362-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:00 smithi183 ceph-mon[33745]: from='client.34688 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi016-74367-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:00 smithi183 ceph-mon[33745]: from='client.34742 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyECPP_smithi016-74781-1", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:20:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:00 smithi183 ceph-mon[33745]: from='client.24685 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi016-74295-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:00 smithi183 ceph-mon[33745]: from='client.24709 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi016-74374-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:00 smithi183 ceph-mon[33745]: from='client.24754 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi016-74692-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:00 smithi183 ceph-mon[33745]: from='client.24691 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIoPP_smithi016-74313-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:00 smithi183 ceph-mon[33745]: from='client.24730 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:00 smithi183 ceph-mon[33745]: from='client.24733 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi016-74514-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:00 smithi183 ceph-mon[33745]: from='client.24748 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi016-74670-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:00 smithi183 ceph-mon[33745]: from='client.24739 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosTierPP_smithi016-74534-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:00 smithi183 ceph-mon[33745]: from='client.24808 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi016-74987-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:00 smithi183 ceph-mon[33745]: from='client.24850 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74884-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:00 smithi183 ceph-mon[33745]: from='client.24859 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74872-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:00 smithi183 ceph-mon[33745]: from='client.24793 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:00.544 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:00 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3950679362' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi016-74781-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi016-74781-1"}]: dispatch 2024-03-26T04:20:00.544 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:00 smithi183 ceph-mon[33745]: osdmap e76: 8 total, 8 up, 8 in 2024-03-26T04:20:00.544 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:00 smithi183 ceph-mon[33745]: from='client.34742 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi016-74781-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi016-74781-1"}]: dispatch 2024-03-26T04:20:00.544 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:00 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:00.544 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:00 smithi183 ceph-mon[33745]: from='client.24901 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi016-75488-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi016-75488-1"}]': finished 2024-03-26T04:20:00.545 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:00 smithi183 ceph-mon[33745]: from='client.24904 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi016-75529-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi016-75529-1"}]': finished 2024-03-26T04:20:00.545 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:00 smithi183 ceph-mon[33745]: osdmap e77: 8 total, 8 up, 8 in 2024-03-26T04:20:00.545 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:00 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3343786834' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi016-74264-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:00.545 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:00 smithi183 ceph-mon[33745]: from='client.35075 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi016-74264-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:00.614 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: 5... 2024-03-26T04:20:00.615 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_15_[76459]: listed object 50... 2024-03-26T04:20:00.615 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_15_[76459]: listed object 75... 2024-03-26T04:20:00.615 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_15_[76459]: listed object 100... 2024-03-26T04:20:00.615 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_15_[76459]: listed object 125... 2024-03-26T04:20:00.615 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_15_[76459]: saw 150 objects 2024-03-26T04:20:00.615 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_15_[76459]: shutting down. 2024-03-26T04:20:00.616 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: ******************************* 2024-03-26T04:20:00.616 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: ******************************* 2024-03-26T04:20:00.616 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: ******************************* 2024-03-26T04:20:00.616 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: ******************************* 2024-03-26T04:20:00.616 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_16_[76460]: starting. 2024-03-26T04:20:00.616 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_16_[76460]: added 25 objects... 2024-03-26T04:20:00.616 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_16_[76460]: added half of the objects 2024-03-26T04:20:00.617 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_16_[76460]: added 50 objects... 2024-03-26T04:20:00.617 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_16_[76460]: added 50 objects 2024-03-26T04:20:00.617 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: process_16_[76460]: shutting down. 2024-03-26T04:20:00.617 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: ******************************* 2024-03-26T04:20:00.617 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: ******************************* 2024-03-26T04:20:00.617 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: ******************************* 2024-03-26T04:20:00.617 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: ******************************* 2024-03-26T04:20:00.617 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: ******************************* 2024-03-26T04:20:00.617 INFO:tasks.workunit.client.0.smithi016.stdout: list_parallel: ******* SUCCESS ********** 2024-03-26T04:20:01.698 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-03-26T04:20:01.699 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-03-26T04:20:01.699 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-03-26T04:20:01.699 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-03-26T04:20:01.699 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-03-26T04:20:01.699 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-03-26T04:20:01.699 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-03-26T04:20:01.699 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (305 ms) 2024-03-26T04:20:01.699 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-03-26T04:20:01.699 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (35 ms) 2024-03-26T04:20:01.699 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-03-26T04:20:01.699 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (25 ms) 2024-03-26T04:20:01.699 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-03-26T04:20:01.699 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (37 ms) 2024-03-26T04:20:01.699 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-03-26T04:20:01.700 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (5 ms) 2024-03-26T04:20:01.700 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-03-26T04:20:01.700 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (5 ms) 2024-03-26T04:20:01.700 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-03-26T04:20:01.700 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (4 ms) 2024-03-26T04:20:01.700 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-03-26T04:20:01.700 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (7 ms) 2024-03-26T04:20:01.700 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-03-26T04:20:01.700 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (6 ms) 2024-03-26T04:20:01.700 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-03-26T04:20:01.700 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (4 ms) 2024-03-26T04:20:01.700 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-03-26T04:20:01.700 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (7 ms) 2024-03-26T04:20:01.700 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-03-26T04:20:01.701 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (7 ms) 2024-03-26T04:20:01.701 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-03-26T04:20:01.701 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (15 ms) 2024-03-26T04:20:01.701 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-03-26T04:20:01.701 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (9 ms) 2024-03-26T04:20:01.701 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-03-26T04:20:01.701 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (8 ms) 2024-03-26T04:20:01.701 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-03-26T04:20:01.701 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (5 ms) 2024-03-26T04:20:01.701 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (485 ms total) 2024-03-26T04:20:01.701 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_read_operations: 2024-03-26T04:20:01.701 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-03-26T04:20:01.701 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (12926 ms total) 2024-03-26T04:20:01.701 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-03-26T04:20:01.706 INFO:tasks.workunit.client.0.smithi016.stdout: handler_error: Running main() from gmock_main.cc 2024-03-26T04:20:01.706 INFO:tasks.workunit.client.0.smithi016.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2024-03-26T04:20:01.706 INFO:tasks.workunit.client.0.smithi016.stdout: handler_error: [----------] Global test environment set-up. 2024-03-26T04:20:01.706 INFO:tasks.workunit.client.0.smithi016.stdout: handler_error: [----------] 1 test from neocls_handler_error 2024-03-26T04:20:01.707 INFO:tasks.workunit.client.0.smithi016.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2024-03-26T04:20:01.707 INFO:tasks.workunit.client.0.smithi016.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (12815 ms) 2024-03-26T04:20:01.707 INFO:tasks.workunit.client.0.smithi016.stdout: handler_error: [----------] 1 test from neocls_handler_error (12815 ms total) 2024-03-26T04:20:01.707 INFO:tasks.workunit.client.0.smithi016.stdout: handler_error: 2024-03-26T04:20:01.707 INFO:tasks.workunit.client.0.smithi016.stdout: handler_error: [----------] Global test environment tear-down 2024-03-26T04:20:01.707 INFO:tasks.workunit.client.0.smithi016.stdout: handler_error: [==========] 1 test from 1 test suite ran. (12815 ms total) 2024-03-26T04:20:01.707 INFO:tasks.workunit.client.0.smithi016.stdout: handler_error: [ PASSED ] 1 test. 2024-03-26T04:20:01.713 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd_pp: Running main() from gmock_main.cc 2024-03-26T04:20:01.713 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-03-26T04:20:01.713 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-03-26T04:20:01.713 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-03-26T04:20:01.713 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-03-26T04:20:01.713 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (51 ms) 2024-03-26T04:20:01.713 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-03-26T04:20:01.714 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (62 ms) 2024-03-26T04:20:01.714 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-03-26T04:20:01.714 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (12861 ms) 2024-03-26T04:20:01.714 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (12974 ms total) 2024-03-26T04:20:01.714 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd_pp: 2024-03-26T04:20:01.714 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-03-26T04:20:01.714 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (12974 ms total) 2024-03-26T04:20:01.714 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-03-26T04:20:01.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:01 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/389421248' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-26T04:20:01.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:01 smithi183 ceph-mon[33745]: Health detail: HEALTH_WARN 20 pool(s) do not have an application enabled 2024-03-26T04:20:01.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:01 smithi183 ceph-mon[33745]: [WRN] POOL_APP_NOT_ENABLED: 20 pool(s) do not have an application enabled 2024-03-26T04:20:01.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:01 smithi183 ceph-mon[33745]: application not enabled on pool 'TooBig_smithi016-74264-1' 2024-03-26T04:20:01.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:01 smithi183 ceph-mon[33745]: application not enabled on pool 'TooBigPP_smithi016-74285-1' 2024-03-26T04:20:01.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:01 smithi183 ceph-mon[33745]: application not enabled on pool 'LibRadosLockPP_smithi016-74362-1' 2024-03-26T04:20:01.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:01 smithi183 ceph-mon[33745]: application not enabled on pool 'LibRadosIo_smithi016-74295-1' 2024-03-26T04:20:01.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:01 smithi183 ceph-mon[33745]: application not enabled on pool 'LibRadosLock_smithi016-74367-1' 2024-03-26T04:20:01.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:01 smithi183 ceph-mon[33745]: application not enabled on pool 'LibRadosIoPP_smithi016-74313-1' 2024-03-26T04:20:01.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:01 smithi183 ceph-mon[33745]: application not enabled on pool 'LibRadosSnapshots_smithi016-74578-1' 2024-03-26T04:20:01.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:01 smithi183 ceph-mon[33745]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-26T04:20:01.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:01 smithi183 ceph-mon[33745]: application not enabled on pool 'LibRadosSnapshotsPP_smithi016-74609-1' 2024-03-26T04:20:01.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:01 smithi183 ceph-mon[33745]: application not enabled on pool 'LibRadosList_smithi016-74374-1' 2024-03-26T04:20:01.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:01 smithi183 ceph-mon[33745]: application not enabled on pool 'test-rados-api-smithi016-74595-1' 2024-03-26T04:20:01.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:01 smithi183 ceph-mon[33745]: application not enabled on pool 'LibRadosWatchNotify_smithi016-74841-1' 2024-03-26T04:20:01.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:01 smithi183 ceph-mon[33745]: application not enabled on pool 'LibRadosMiscPP_smithi016-74514-1' 2024-03-26T04:20:01.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:01 smithi183 ceph-mon[33745]: application not enabled on pool 'test-rados-api-smithi016-75032-1' 2024-03-26T04:20:01.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:01 smithi183 ceph-mon[33745]: application not enabled on pool 'LibRadosTierPP_smithi016-74534-1' 2024-03-26T04:20:01.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:01 smithi183 ceph-mon[33745]: application not enabled on pool 'DNEsmithi016-75175-1' 2024-03-26T04:20:01.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:01 smithi183 ceph-mon[33745]: application not enabled on pool 'LibRadosStat_smithi016-74670-1' 2024-03-26T04:20:01.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:01 smithi183 ceph-mon[33745]: application not enabled on pool 'Versionsmithi016-75564-1' 2024-03-26T04:20:01.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:01 smithi183 ceph-mon[33745]: application not enabled on pool 'LibRadosStatPP_smithi016-74692-1' 2024-03-26T04:20:01.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:01 smithi183 ceph-mon[33745]: application not enabled on pool 'SetOpFlagssmithi016-75561-1' 2024-03-26T04:20:01.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:01 smithi183 ceph-mon[33745]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-26T04:20:01.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:01 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:01.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:01 smithi183 ceph-mon[33745]: from='client.34742 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi016-74781-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi016-74781-1"}]': finished 2024-03-26T04:20:01.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:01 smithi183 ceph-mon[33745]: from='client.35075 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi016-74264-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:01.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:01 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2172239581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:01.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:01 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.0"}]: dispatch 2024-03-26T04:20:01.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:01 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.1"}]: dispatch 2024-03-26T04:20:01.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:01 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.2"}]: dispatch 2024-03-26T04:20:01.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:01 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.3"}]: dispatch 2024-03-26T04:20:01.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:01 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.4"}]: dispatch 2024-03-26T04:20:01.795 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:01 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.5"}]: dispatch 2024-03-26T04:20:01.795 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:01 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.6"}]: dispatch 2024-03-26T04:20:01.795 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:01 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.7"}]: dispatch 2024-03-26T04:20:01.795 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:01 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.8"}]: dispatch 2024-03-26T04:20:01.795 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:01 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.9"}]: dispatch 2024-03-26T04:20:01.795 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:01 smithi183 ceph-mon[33745]: osdmap e78: 8 total, 8 up, 8 in 2024-03-26T04:20:01.795 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:01 smithi183 ceph-mon[33745]: from='client.35156 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:01.819 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: Running main() from gmock_main.cc 2024-03-26T04:20:01.819 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-03-26T04:20:01.819 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: [----------] Global test environment set-up. 2024-03-26T04:20:01.819 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-03-26T04:20:01.819 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-03-26T04:20:01.819 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (76 ms) 2024-03-26T04:20:01.819 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-03-26T04:20:01.819 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (47 ms) 2024-03-26T04:20:01.819 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-03-26T04:20:01.820 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (12847 ms) 2024-03-26T04:20:01.820 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-03-26T04:20:01.820 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:00.066921+0000 mon.a [WRN] Health detail: HEALTH_WARN 20 pool(s) do not have an application enabled 2024-03-26T04:20:01.820 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:00.066943+0000 mon.a [WRN] [WRN] POOL_APP_NOT_ENABLED: 20 pool(s) do not have an application enabled 2024-03-26T04:20:01.820 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:00.066951+0000 mon.a [WRN] application not enabled on pool 'TooBig_smithi016-74264-1' 2024-03-26T04:20:01.820 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:00.066965+0000 mon.a [WRN] application not enabled on pool 'TooBigPP_smithi016-74285-1' 2024-03-26T04:20:01.820 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:00.066974+0000 mon.a [WRN] application not enabled on pool 'LibRadosLockPP_smithi016-74362-1' 2024-03-26T04:20:01.820 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:00.066981+0000 mon.a [WRN] application not enabled on pool 'LibRadosIo_smithi016-74295-1' 2024-03-26T04:20:01.820 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:00.066990+0000 mon.a [WRN] application not enabled on pool 'LibRadosLock_smithi016-74367-1' 2024-03-26T04:20:01.820 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:00.067000+0000 mon.a [WRN] application not enabled on pool 'LibRadosIoPP_smithi016-74313-1' 2024-03-26T04:20:01.820 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:00.067010+0000 mon.a [WRN] application not enabled on pool 'LibRadosSnapshots_smithi016-74578-1' 2024-03-26T04:20:01.820 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:00.067030+0000 mon.a [WRN] application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-26T04:20:01.820 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:00.067042+0000 mon.a [WRN] application not enabled on pool 'LibRadosSnapshotsPP_smithi016-74609-1' 2024-03-26T04:20:01.820 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:00.067052+0000 mon.a [WRN] application not enabled on pool 'LibRadosList_smithi016-74374-1' 2024-03-26T04:20:01.821 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:00.067062+0000 mon.a [WRN] application not enabled on pool 'test-rados-api-smithi016-74595-1' 2024-03-26T04:20:01.821 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:00.067083+0000 mon.a [WRN] application not enabled on pool 'LibRadosWatchNotify_smithi016-74841-1' 2024-03-26T04:20:01.821 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:00.067103+0000 mon.a [WRN] application not enabled on pool 'LibRadosMiscPP_smithi016-74514-1' 2024-03-26T04:20:01.821 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:00.067116+0000 mon.a [WRN] application not enabled on pool 'test-rados-api-smithi016-75032-1' 2024-03-26T04:20:01.821 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:00.067131+0000 mon.a [WRN] application not enabled on pool 'LibRadosTierPP_smithi016-74534-1' 2024-03-26T04:20:01.821 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:00.067143+0000 mon.a [WRN] application not enabled on pool 'DNEsmithi016-75175-1' 2024-03-26T04:20:01.821 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:00.067161+0000 mon.a [WRN] application not enabled on pool 'LibRadosStat_smithi016-74670-1' 2024-03-26T04:20:01.821 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:00.067172+0000 mon.a [WRN] application not enabled on pool 'Versionsmithi016-75564-1' 2024-03-26T04:20:01.821 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:00.067184+0000 mon.a [WRN] application not enabled on pool 'LibRadosStatPP_smithi016-74692-1' 2024-03-26T04:20:01.821 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:00.067200+0000 mon.a [WRN] application not enabled on pool 'SetOpFlagssmithi016-75561-1' 2024-03-26T04:20:01.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/389421248' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-26T04:20:01.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[37215]: Health detail: HEALTH_WARN 20 pool(s) do not have an application enabled 2024-03-26T04:20:01.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[37215]: [WRN] POOL_APP_NOT_ENABLED: 20 pool(s) do not have an application enabled 2024-03-26T04:20:01.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[37215]: application not enabled on pool 'TooBig_smithi016-74264-1' 2024-03-26T04:20:01.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[37215]: application not enabled on pool 'TooBigPP_smithi016-74285-1' 2024-03-26T04:20:01.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[37215]: application not enabled on pool 'LibRadosLockPP_smithi016-74362-1' 2024-03-26T04:20:01.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[37215]: application not enabled on pool 'LibRadosIo_smithi016-74295-1' 2024-03-26T04:20:01.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[37215]: application not enabled on pool 'LibRadosLock_smithi016-74367-1' 2024-03-26T04:20:01.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[37215]: application not enabled on pool 'LibRadosIoPP_smithi016-74313-1' 2024-03-26T04:20:01.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[37215]: application not enabled on pool 'LibRadosSnapshots_smithi016-74578-1' 2024-03-26T04:20:01.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[37215]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-26T04:20:01.917 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[37215]: application not enabled on pool 'LibRadosSnapshotsPP_smithi016-74609-1' 2024-03-26T04:20:01.917 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[37215]: application not enabled on pool 'LibRadosList_smithi016-74374-1' 2024-03-26T04:20:01.917 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[37215]: application not enabled on pool 'test-rados-api-smithi016-74595-1' 2024-03-26T04:20:01.917 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[37215]: application not enabled on pool 'LibRadosWatchNotify_smithi016-74841-1' 2024-03-26T04:20:01.917 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[37215]: application not enabled on pool 'LibRadosMiscPP_smithi016-74514-1' 2024-03-26T04:20:01.917 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[37215]: application not enabled on pool 'test-rados-api-smithi016-75032-1' 2024-03-26T04:20:01.917 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[37215]: application not enabled on pool 'LibRadosTierPP_smithi016-74534-1' 2024-03-26T04:20:01.917 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[37215]: application not enabled on pool 'DNEsmithi016-75175-1' 2024-03-26T04:20:01.917 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[37215]: application not enabled on pool 'LibRadosStat_smithi016-74670-1' 2024-03-26T04:20:01.917 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[37215]: application not enabled on pool 'Versionsmithi016-75564-1' 2024-03-26T04:20:01.917 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[37215]: application not enabled on pool 'LibRadosStatPP_smithi016-74692-1' 2024-03-26T04:20:01.917 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[37215]: application not enabled on pool 'SetOpFlagssmithi016-75561-1' 2024-03-26T04:20:01.917 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[37215]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-26T04:20:01.917 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:01.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[37215]: from='client.34742 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi016-74781-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi016-74781-1"}]': finished 2024-03-26T04:20:01.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[37215]: from='client.35075 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi016-74264-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:01.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2172239581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:01.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.0"}]: dispatch 2024-03-26T04:20:01.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.1"}]: dispatch 2024-03-26T04:20:01.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.2"}]: dispatch 2024-03-26T04:20:01.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.3"}]: dispatch 2024-03-26T04:20:01.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.4"}]: dispatch 2024-03-26T04:20:01.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.5"}]: dispatch 2024-03-26T04:20:01.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.6"}]: dispatch 2024-03-26T04:20:01.919 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.7"}]: dispatch 2024-03-26T04:20:01.919 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.8"}]: dispatch 2024-03-26T04:20:01.919 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.9"}]: dispatch 2024-03-26T04:20:01.919 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[37215]: osdmap e78: 8 total, 8 up, 8 in 2024-03-26T04:20:01.919 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[37215]: from='client.35156 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:01.919 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/389421248' entity='client.admin' cmd=[{"prefix":"status"}]: dispatch 2024-03-26T04:20:01.919 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[29101]: Health detail: HEALTH_WARN 20 pool(s) do not have an application enabled 2024-03-26T04:20:01.919 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[29101]: [WRN] POOL_APP_NOT_ENABLED: 20 pool(s) do not have an application enabled 2024-03-26T04:20:01.919 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[29101]: application not enabled on pool 'TooBig_smithi016-74264-1' 2024-03-26T04:20:01.920 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[29101]: application not enabled on pool 'TooBigPP_smithi016-74285-1' 2024-03-26T04:20:01.920 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[29101]: application not enabled on pool 'LibRadosLockPP_smithi016-74362-1' 2024-03-26T04:20:01.920 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[29101]: application not enabled on pool 'LibRadosIo_smithi016-74295-1' 2024-03-26T04:20:01.920 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[29101]: application not enabled on pool 'LibRadosLock_smithi016-74367-1' 2024-03-26T04:20:01.920 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[29101]: application not enabled on pool 'LibRadosIoPP_smithi016-74313-1' 2024-03-26T04:20:01.920 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[29101]: application not enabled on pool 'LibRadosSnapshots_smithi016-74578-1' 2024-03-26T04:20:01.920 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[29101]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-26T04:20:01.920 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[29101]: application not enabled on pool 'LibRadosSnapshotsPP_smithi016-74609-1' 2024-03-26T04:20:01.920 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[29101]: application not enabled on pool 'LibRadosList_smithi016-74374-1' 2024-03-26T04:20:01.920 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[29101]: application not enabled on pool 'test-rados-api-smithi016-74595-1' 2024-03-26T04:20:01.920 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[29101]: application not enabled on pool 'LibRadosWatchNotify_smithi016-74841-1' 2024-03-26T04:20:01.920 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[29101]: application not enabled on pool 'LibRadosMiscPP_smithi016-74514-1' 2024-03-26T04:20:01.920 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[29101]: application not enabled on pool 'test-rados-api-smithi016-75032-1' 2024-03-26T04:20:01.920 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[29101]: application not enabled on pool 'LibRadosTierPP_smithi016-74534-1' 2024-03-26T04:20:01.921 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[29101]: application not enabled on pool 'DNEsmithi016-75175-1' 2024-03-26T04:20:01.921 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[29101]: application not enabled on pool 'LibRadosStat_smithi016-74670-1' 2024-03-26T04:20:01.921 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[29101]: application not enabled on pool 'Versionsmithi016-75564-1' 2024-03-26T04:20:01.921 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[29101]: application not enabled on pool 'LibRadosStatPP_smithi016-74692-1' 2024-03-26T04:20:01.921 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[29101]: application not enabled on pool 'SetOpFlagssmithi016-75561-1' 2024-03-26T04:20:01.921 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[29101]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-26T04:20:01.921 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:01.921 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[29101]: from='client.34742 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi016-74781-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi016-74781-1"}]': finished 2024-03-26T04:20:01.921 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[29101]: from='client.35075 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi016-74264-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:01.921 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2172239581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:01.921 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.0"}]: dispatch 2024-03-26T04:20:01.921 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.1"}]: dispatch 2024-03-26T04:20:01.921 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.2"}]: dispatch 2024-03-26T04:20:01.922 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.3"}]: dispatch 2024-03-26T04:20:01.922 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.4"}]: dispatch 2024-03-26T04:20:01.922 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.5"}]: dispatch 2024-03-26T04:20:01.922 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.6"}]: dispatch 2024-03-26T04:20:01.922 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.7"}]: dispatch 2024-03-26T04:20:01.923 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.8"}]: dispatch 2024-03-26T04:20:01.923 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.9"}]: dispatch 2024-03-26T04:20:01.923 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[29101]: osdmap e78: 8 total, 8 up, 8 in 2024-03-26T04:20:01.923 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:01 smithi016 ceph-mon[29101]: from='client.35156 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:02.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:20:02 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:20:02.533 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:00.067213+0000 mon.a [WRN] use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-26T04:20:02.533 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:00.605341+0000 mon.a [INF] from='client.34742 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi016-74781-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi016-74781-1"}]': finished 2024-03-26T04:20:02.534 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:00.605504+0000 mon.a [INF] from='client.35075 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi016-74264-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:02.534 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:00.612956+0000 mon.c [INF] from='client.? v1:172.21.15.16:0/2172239581' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:02.534 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:00.879513+0000 mon.b [INF] from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.0"}]: dispatch 2024-03-26T04:20:02.534 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:00.880126+0000 mon.b [INF] from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.1"}]: dispatch 2024-03-26T04:20:02.534 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:00.880541+0000 mon.b [INF] from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.2"}]: dispatch 2024-03-26T04:20:02.534 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:00.880995+0000 mon.b [INF] from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.3"}]: dispatch 2024-03-26T04:20:02.534 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:00.881415+0000 mon.b [INF] from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.4"}]: dispatch 2024-03-26T04:20:02.534 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:00.881819+0000 mon.b [INF] from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.5"}]: dispatch 2024-03-26T04:20:02.534 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:00.882233+0000 mon.b [INF] from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.6"}]: dispatch 2024-03-26T04:20:02.534 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:00.882629+0000 mon.b [INF] from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.7"}]: dispatch 2024-03-26T04:20:02.534 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:00.883028+0000 mon.b [INF] from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.8"}]: dispatch 2024-03-26T04:20:02.534 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:00.883400+0000 mon.b [INF] from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.9"}]: dispatch 2024-03-26T04:20:02.535 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:00.887839+0000 mon.a [INF] from='client.35156 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:02.535 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:01.685430+0000 mon.a [INF] from='client.35156 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:02.535 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:01.694738+0000 mon.b [INF] from='client.? v1:172.21.15.16:0/1973575539' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:02.535 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:01.695265+0000 mon.b [INF] from='client.? v1:172.21.15.16:0/3264354453' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi016-75488-1"}]: dispatch 2024-03-26T04:20:02.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:02 smithi183 ceph-mon[33745]: pgmap v73: 1220 pgs: 296 creating+peering, 132 active+clean, 792 unknown; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:20:02.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:02 smithi183 ceph-mon[33745]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.0"}]: dispatch 2024-03-26T04:20:02.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:02 smithi183 ceph-mon[33745]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.1"}]: dispatch 2024-03-26T04:20:02.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:02 smithi183 ceph-mon[33745]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.2"}]: dispatch 2024-03-26T04:20:02.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:02 smithi183 ceph-mon[33745]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.3"}]: dispatch 2024-03-26T04:20:02.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:02 smithi183 ceph-mon[33745]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.4"}]: dispatch 2024-03-26T04:20:02.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:02 smithi183 ceph-mon[33745]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.5"}]: dispatch 2024-03-26T04:20:02.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:02 smithi183 ceph-mon[33745]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.6"}]: dispatch 2024-03-26T04:20:02.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:02 smithi183 ceph-mon[33745]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.7"}]: dispatch 2024-03-26T04:20:02.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:02 smithi183 ceph-mon[33745]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.8"}]: dispatch 2024-03-26T04:20:02.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:02 smithi183 ceph-mon[33745]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.9"}]: dispatch 2024-03-26T04:20:02.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:02 smithi183 ceph-mon[33745]: pgmap v76: 1116 pgs: 200 creating+peering, 132 active+clean, 784 unknown; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:20:02.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:02 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:02.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:02 smithi183 ceph-mon[33745]: 19.6 deep-scrub starts 2024-03-26T04:20:02.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:02 smithi183 ceph-mon[33745]: 19.6 deep-scrub ok 2024-03-26T04:20:02.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:02 smithi183 ceph-mon[33745]: 19.0 deep-scrub starts 2024-03-26T04:20:02.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:02 smithi183 ceph-mon[33745]: 19.0 deep-scrub ok 2024-03-26T04:20:02.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:02 smithi183 ceph-mon[33745]: from='client.35156 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:02.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:02 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1973575539' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:02.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:02 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3264354453' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi016-75488-1"}]: dispatch 2024-03-26T04:20:02.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:02 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2267327916' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi016-74692-7"}]: dispatch 2024-03-26T04:20:02.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:02 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/806805394' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi016-74313-22"}]: dispatch 2024-03-26T04:20:02.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:02 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1385062143' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi016-74670-7"}]: dispatch 2024-03-26T04:20:02.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:02 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3019850196' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi016-74362-10"}]: dispatch 2024-03-26T04:20:02.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:02 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3172019969' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi016-74295-16"}]: dispatch 2024-03-26T04:20:02.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:02 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/883729375' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-26T04:20:02.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:02 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1871982092' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi016-74367-10"}]: dispatch 2024-03-26T04:20:02.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:02 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:02.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:02 smithi183 ceph-mon[33745]: osdmap e79: 8 total, 8 up, 8 in 2024-03-26T04:20:02.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:02 smithi183 ceph-mon[33745]: from='client.25081 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:02.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:02 smithi183 ceph-mon[33745]: from='client.24901 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi016-75488-1"}]: dispatch 2024-03-26T04:20:02.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[29101]: pgmap v73: 1220 pgs: 296 creating+peering, 132 active+clean, 792 unknown; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:20:02.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[29101]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.0"}]: dispatch 2024-03-26T04:20:02.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[29101]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.1"}]: dispatch 2024-03-26T04:20:02.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[29101]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.2"}]: dispatch 2024-03-26T04:20:02.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[29101]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.3"}]: dispatch 2024-03-26T04:20:02.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[29101]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.4"}]: dispatch 2024-03-26T04:20:02.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[29101]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.5"}]: dispatch 2024-03-26T04:20:02.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[29101]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.6"}]: dispatch 2024-03-26T04:20:02.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[29101]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.7"}]: dispatch 2024-03-26T04:20:02.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[29101]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.8"}]: dispatch 2024-03-26T04:20:02.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[29101]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.9"}]: dispatch 2024-03-26T04:20:02.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[29101]: pgmap v76: 1116 pgs: 200 creating+peering, 132 active+clean, 784 unknown; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:20:02.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:02.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[29101]: 19.6 deep-scrub starts 2024-03-26T04:20:02.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[29101]: 19.6 deep-scrub ok 2024-03-26T04:20:02.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[29101]: 19.0 deep-scrub starts 2024-03-26T04:20:02.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[29101]: 19.0 deep-scrub ok 2024-03-26T04:20:02.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[29101]: from='client.35156 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:02.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1973575539' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:02.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3264354453' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi016-75488-1"}]: dispatch 2024-03-26T04:20:02.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2267327916' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi016-74692-7"}]: dispatch 2024-03-26T04:20:02.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/806805394' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi016-74313-22"}]: dispatch 2024-03-26T04:20:02.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1385062143' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi016-74670-7"}]: dispatch 2024-03-26T04:20:02.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3019850196' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi016-74362-10"}]: dispatch 2024-03-26T04:20:02.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[37215]: pgmap v73: 1220 pgs: 296 creating+peering, 132 active+clean, 792 unknown; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:20:02.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[37215]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.0"}]: dispatch 2024-03-26T04:20:02.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[37215]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.1"}]: dispatch 2024-03-26T04:20:02.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[37215]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.2"}]: dispatch 2024-03-26T04:20:02.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[37215]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.3"}]: dispatch 2024-03-26T04:20:02.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[37215]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.4"}]: dispatch 2024-03-26T04:20:02.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[37215]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.5"}]: dispatch 2024-03-26T04:20:02.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[37215]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.6"}]: dispatch 2024-03-26T04:20:02.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[37215]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.7"}]: dispatch 2024-03-26T04:20:02.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[37215]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.8"}]: dispatch 2024-03-26T04:20:02.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[37215]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.9"}]: dispatch 2024-03-26T04:20:02.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[37215]: pgmap v76: 1116 pgs: 200 creating+peering, 132 active+clean, 784 unknown; 582 KiB data, 243 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:20:02.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:02.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[37215]: 19.6 deep-scrub starts 2024-03-26T04:20:02.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[37215]: 19.6 deep-scrub ok 2024-03-26T04:20:02.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[37215]: 19.0 deep-scrub starts 2024-03-26T04:20:02.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[37215]: 19.0 deep-scrub ok 2024-03-26T04:20:02.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[37215]: from='client.35156 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:02.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1973575539' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:02.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3264354453' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi016-75488-1"}]: dispatch 2024-03-26T04:20:02.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2267327916' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi016-74692-7"}]: dispatch 2024-03-26T04:20:02.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/806805394' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi016-74313-22"}]: dispatch 2024-03-26T04:20:02.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1385062143' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi016-74670-7"}]: dispatch 2024-03-26T04:20:02.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3019850196' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi016-74362-10"}]: dispatch 2024-03-26T04:20:02.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3172019969' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi016-74295-16"}]: dispatch 2024-03-26T04:20:02.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/883729375' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-26T04:20:02.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1871982092' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi016-74367-10"}]: dispatch 2024-03-26T04:20:02.917 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:02.917 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[37215]: osdmap e79: 8 total, 8 up, 8 in 2024-03-26T04:20:02.917 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[37215]: from='client.25081 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:02.917 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[37215]: from='client.24901 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi016-75488-1"}]: dispatch 2024-03-26T04:20:02.917 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3172019969' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi016-74295-16"}]: dispatch 2024-03-26T04:20:02.917 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/883729375' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-26T04:20:02.917 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1871982092' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi016-74367-10"}]: dispatch 2024-03-26T04:20:02.917 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:02.917 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[29101]: osdmap e79: 8 total, 8 up, 8 in 2024-03-26T04:20:02.917 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[29101]: from='client.25081 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:02.918 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:02 smithi016 ceph-mon[29101]: from='client.24901 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi016-75488-1"}]: dispatch 2024-03-26T04:20:03.581 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:01.786111+0000 mon.b [INF] from='client.? v1:172.21.15.16:0/2267327916' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi016-74692-7"}]: dispatch 2024-03-26T04:20:03.581 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:01.801670+0000 mon.b [INF] from='client.? v1:172.21.15.16:0/806805394' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi016-74313-22"}]: dispatch 2024-03-26T04:20:03.581 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:01.803041+0000 mon.b [INF] from='client.? v1:172.21.15.16:0/1385062143' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi016-74670-7"}]: dispatch 2024-03-26T04:20:03.581 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:01.805195+0000 mon.b [INF] from='client.? v1:172.21.15.16:0/3019850196' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi016-74362-10"}]: dispatch 2024-03-26T04:20:03.581 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:01.811084+0000 mon.b [INF] from='client.? v1:172.21.15.16:0/3172019969' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi016-74295-16"}]: dispatch 2024-03-26T04:20:03.581 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:01.813246+0000 mon.c [INF] from='client.? v1:172.21.15.16:0/883729375' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-26T04:20:03.581 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:01.844038+0000 mon.b [INF] from='client.? v1:172.21.15.16:0/1871982092' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi016-74367-10"}]: dispatch 2024-03-26T04:20:03.581 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:02.277262+0000 mon.a [INF] from='client.25081 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:03.581 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:02.278348+0000 mon.a [INF] from='client.24901 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi016-75488-1"}]: dispatch 2024-03-26T04:20:03.581 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:01.813640+0000 client.admin [INF] onexx 2024-03-26T04:20:03.582 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:02.531297+0000 mon.a [INF] from='client.35222 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-26T04:20:03.582 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:02.537565+0000 mon.a [INF] from='client.25270 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi016-74692-7"}]: dispatch 2024-03-26T04:20:03.582 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:02.538161+0000 mon.b [INF] from='client.? v1:172.21.15.16:0/2267327916' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi016-74692-7"}]: dispatch 2024-03-26T04:20:03.582 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:02.793672+0000 mon.a [INF] from='client.25081 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:03.582 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:02.793760+0000 mon.a [INF] from='client.24901 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi016-75488-1"}]': finished 2024-03-26T04:20:03.582 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:02.799049+0000 mon.c [INF] from='client.? v1:172.21.15.16:0/984684998' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:03.582 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:02.799274+0000 mon.b [INF] from='client.? v1:172.21.15.16:0/3264354453' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi016-75488-1"}]: dispatch 2024-03-26T04:20:03.858 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:02.800402+0000 mon.b [INF] from='client.? v1:172.21.15.16:0/1757055001' entity='client.admin' cmd=[{"prefix": "osd p cls: Running main() from gmock_main.cc 2024-03-26T04:20:03.858 INFO:tasks.workunit.client.0.smithi016.stdout: cls: [==========] Running 2 tests from 1 test suite. 2024-03-26T04:20:03.858 INFO:tasks.workunit.client.0.smithi016.stdout: cls: [----------] Global test environment set-up. 2024-03-26T04:20:03.858 INFO:tasks.workunit.client.0.smithi016.stdout: cls: [----------] 2 tests from NeoRadosCls 2024-03-26T04:20:03.858 INFO:tasks.workunit.client.0.smithi016.stdout: cls: [ RUN ] NeoRadosCls.DNE 2024-03-26T04:20:03.859 INFO:tasks.workunit.client.0.smithi016.stdout: cls: [ OK ] NeoRadosCls.DNE (10914 ms) 2024-03-26T04:20:03.859 INFO:tasks.workunit.client.0.smithi016.stdout: cls: [ RUN ] NeoRadosCls.RemoteReads 2024-03-26T04:20:03.859 INFO:tasks.workunit.client.0.smithi016.stdout: cls: [ OK ] NeoRadosCls.RemoteReads (4097 ms) 2024-03-26T04:20:03.859 INFO:tasks.workunit.client.0.smithi016.stdout: cls: [----------] 2 tests from NeoRadosCls (15011 ms total) 2024-03-26T04:20:03.859 INFO:tasks.workunit.client.0.smithi016.stdout: cls: 2024-03-26T04:20:03.859 INFO:tasks.workunit.client.0.smithi016.stdout: cls: [----------] Global test environment tear-down 2024-03-26T04:20:03.859 INFO:tasks.workunit.client.0.smithi016.stdout: cls: [==========] 2 tests from 1 test suite ran. (15012 ms total) 2024-03-26T04:20:03.859 INFO:tasks.workunit.client.0.smithi016.stdout: cls: [ PASSED ] 2 tests. 2024-03-26T04:20:03.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:03 smithi016 ceph-mon[37215]: onexx 2024-03-26T04:20:03.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:03 smithi016 ceph-mon[37215]: from='client.35222 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-26T04:20:03.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:03 smithi016 ceph-mon[37215]: from='client.25270 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi016-74692-7"}]: dispatch 2024-03-26T04:20:03.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:03 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2267327916' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi016-74692-7"}]: dispatch 2024-03-26T04:20:03.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:03 smithi016 ceph-mon[37215]: from='client.25081 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:03.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:03 smithi016 ceph-mon[37215]: from='client.24901 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi016-75488-1"}]': finished 2024-03-26T04:20:03.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:03 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/984684998' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:03.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:03 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3264354453' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi016-75488-1"}]: dispatch 2024-03-26T04:20:03.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:03 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1757055001' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi016-74285-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:03.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:03 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:03.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:03 smithi016 ceph-mon[37215]: osdmap e80: 8 total, 8 up, 8 in 2024-03-26T04:20:03.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:03 smithi016 ceph-mon[37215]: from='client.25288 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi016-74313-22"}]: dispatch 2024-03-26T04:20:03.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:03 smithi016 ceph-mon[37215]: from='client.35192 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:03.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:03 smithi016 ceph-mon[37215]: from='client.25282 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi016-74670-7"}]: dispatch 2024-03-26T04:20:03.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:03 smithi016 ceph-mon[29101]: onexx 2024-03-26T04:20:03.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:03 smithi016 ceph-mon[29101]: from='client.35222 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-26T04:20:03.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:03 smithi016 ceph-mon[29101]: from='client.25270 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi016-74692-7"}]: dispatch 2024-03-26T04:20:03.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:03 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2267327916' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi016-74692-7"}]: dispatch 2024-03-26T04:20:03.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:03 smithi016 ceph-mon[29101]: from='client.25081 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:03.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:03 smithi016 ceph-mon[29101]: from='client.24901 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi016-75488-1"}]': finished 2024-03-26T04:20:03.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:03 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/984684998' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:03.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:03 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3264354453' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi016-75488-1"}]: dispatch 2024-03-26T04:20:03.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:03 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1757055001' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi016-74285-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:03.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:03 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:03.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:03 smithi016 ceph-mon[29101]: osdmap e80: 8 total, 8 up, 8 in 2024-03-26T04:20:03.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:03 smithi016 ceph-mon[29101]: from='client.25288 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi016-74313-22"}]: dispatch 2024-03-26T04:20:03.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:03 smithi016 ceph-mon[29101]: from='client.35192 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:03.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:03 smithi016 ceph-mon[29101]: from='client.25282 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi016-74670-7"}]: dispatch 2024-03-26T04:20:04.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:03 smithi183 ceph-mon[33745]: onexx 2024-03-26T04:20:04.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:03 smithi183 ceph-mon[33745]: from='client.35222 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-03-26T04:20:04.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:03 smithi183 ceph-mon[33745]: from='client.25270 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi016-74692-7"}]: dispatch 2024-03-26T04:20:04.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:03 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2267327916' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi016-74692-7"}]: dispatch 2024-03-26T04:20:04.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:03 smithi183 ceph-mon[33745]: from='client.25081 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:04.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:03 smithi183 ceph-mon[33745]: from='client.24901 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi016-75488-1"}]': finished 2024-03-26T04:20:04.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:03 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/984684998' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:04.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:03 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3264354453' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi016-75488-1"}]: dispatch 2024-03-26T04:20:04.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:03 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1757055001' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi016-74285-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:04.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:03 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:04.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:03 smithi183 ceph-mon[33745]: osdmap e80: 8 total, 8 up, 8 in 2024-03-26T04:20:04.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:03 smithi183 ceph-mon[33745]: from='client.25288 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi016-74313-22"}]: dispatch 2024-03-26T04:20:04.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:03 smithi183 ceph-mon[33745]: from='client.35192 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:04.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:03 smithi183 ceph-mon[33745]: from='client.25282 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi016-74670-7"}]: dispatch 2024-03-26T04:20:04.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[29101]: 19.9 deep-scrub starts 2024-03-26T04:20:04.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[29101]: 19.9 deep-scrub ok 2024-03-26T04:20:04.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[29101]: 19.7 deep-scrub starts 2024-03-26T04:20:04.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[29101]: 19.7 deep-scrub ok 2024-03-26T04:20:04.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[29101]: 19.3 deep-scrub starts 2024-03-26T04:20:04.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[29101]: 19.3 deep-scrub ok 2024-03-26T04:20:04.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:20:04.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[29101]: 19.5 deep-scrub starts 2024-03-26T04:20:04.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[29101]: 19.5 deep-scrub ok 2024-03-26T04:20:04.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/806805394' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi016-74313-22"}]: dispatch 2024-03-26T04:20:04.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1385062143' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi016-74670-7"}]: dispatch 2024-03-26T04:20:04.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[29101]: from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi016-74362-10"}]: dispatch 2024-03-26T04:20:04.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[29101]: from='client.35222 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-03-26T04:20:04.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3019850196' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi016-74362-10"}]: dispatch 2024-03-26T04:20:04.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/883729375' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-26T04:20:04.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[29101]: twoxx 2024-03-26T04:20:04.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[29101]: from='client.35222 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-26T04:20:04.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[29101]: from='client.25294 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi016-74295-16"}]: dispatch 2024-03-26T04:20:04.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3172019969' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi016-74295-16"}]: dispatch 2024-03-26T04:20:04.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[29101]: from='client.25306 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi016-74367-10"}]: dispatch 2024-03-26T04:20:04.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1871982092' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi016-74367-10"}]: dispatch 2024-03-26T04:20:04.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[29101]: from='client.25270 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi016-74692-7"}]: dispatch 2024-03-26T04:20:04.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[29101]: from='client.24901 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi016-75488-1"}]: dispatch 2024-03-26T04:20:04.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[29101]: from='client.25210 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi016-74285-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:04.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2267327916' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi016-74692-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:04.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[29101]: from='client.35192 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:04.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[29101]: from='client.24901 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi016-75488-1"}]': finished 2024-03-26T04:20:04.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[29101]: from='client.25210 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi016-74285-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:04.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3950679362' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi016-74781-1"}]: dispatch 2024-03-26T04:20:04.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/287729995' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi016-74508-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:04.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2219155529' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi016-75529-1"}]: dispatch 2024-03-26T04:20:04.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1973575539' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:04.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3740118015' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi016-75488-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:04.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:04.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[29101]: osdmap e81: 8 total, 8 up, 8 in 2024-03-26T04:20:04.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[29101]: Health check update: 16 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:20:04.917 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[29101]: from='client.34742 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi016-74781-1"}]: dispatch 2024-03-26T04:20:04.917 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[29101]: from='client.35246 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi016-74508-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:04.917 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[29101]: from='client.35264 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi016-75488-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:04.917 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[37215]: 19.9 deep-scrub starts 2024-03-26T04:20:04.917 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[37215]: 19.9 deep-scrub ok 2024-03-26T04:20:04.917 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[37215]: 19.7 deep-scrub starts 2024-03-26T04:20:04.917 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[37215]: 19.7 deep-scrub ok 2024-03-26T04:20:04.917 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[37215]: 19.3 deep-scrub starts 2024-03-26T04:20:04.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[37215]: 19.3 deep-scrub ok 2024-03-26T04:20:04.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:20:04.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[37215]: 19.5 deep-scrub starts 2024-03-26T04:20:04.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[37215]: 19.5 deep-scrub ok 2024-03-26T04:20:04.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/806805394' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi016-74313-22"}]: dispatch 2024-03-26T04:20:04.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1385062143' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi016-74670-7"}]: dispatch 2024-03-26T04:20:04.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[37215]: from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi016-74362-10"}]: dispatch 2024-03-26T04:20:04.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[37215]: from='client.35222 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-03-26T04:20:04.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3019850196' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi016-74362-10"}]: dispatch 2024-03-26T04:20:04.919 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/883729375' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-26T04:20:04.919 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[37215]: twoxx 2024-03-26T04:20:04.919 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[37215]: from='client.35222 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-26T04:20:04.919 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[37215]: from='client.25294 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi016-74295-16"}]: dispatch 2024-03-26T04:20:04.919 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3172019969' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi016-74295-16"}]: dispatch 2024-03-26T04:20:04.919 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[37215]: from='client.25306 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi016-74367-10"}]: dispatch 2024-03-26T04:20:04.919 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1871982092' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi016-74367-10"}]: dispatch 2024-03-26T04:20:04.919 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[37215]: from='client.25270 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi016-74692-7"}]: dispatch 2024-03-26T04:20:04.919 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[37215]: from='client.24901 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi016-75488-1"}]: dispatch 2024-03-26T04:20:04.919 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[37215]: from='client.25210 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi016-74285-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:04.920 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2267327916' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi016-74692-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:04.920 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[37215]: from='client.35192 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:04.920 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[37215]: from='client.24901 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi016-75488-1"}]': finished 2024-03-26T04:20:04.920 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[37215]: from='client.25210 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi016-74285-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:04.920 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3950679362' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi016-74781-1"}]: dispatch 2024-03-26T04:20:04.920 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/287729995' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi016-74508-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:04.920 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2219155529' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi016-75529-1"}]: dispatch 2024-03-26T04:20:04.920 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1973575539' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:04.920 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3740118015' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi016-75488-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:04.920 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:04.920 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[37215]: osdmap e81: 8 total, 8 up, 8 in 2024-03-26T04:20:04.920 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[37215]: Health check update: 16 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:20:04.921 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[37215]: from='client.34742 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi016-74781-1"}]: dispatch 2024-03-26T04:20:04.921 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[37215]: from='client.35246 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi016-74508-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:04.921 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:04 smithi016 ceph-mon[37215]: from='client.35264 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi016-75488-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:05.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:04 smithi183 ceph-mon[33745]: 19.9 deep-scrub starts 2024-03-26T04:20:05.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:04 smithi183 ceph-mon[33745]: 19.9 deep-scrub ok 2024-03-26T04:20:05.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:04 smithi183 ceph-mon[33745]: 19.7 deep-scrub starts 2024-03-26T04:20:05.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:04 smithi183 ceph-mon[33745]: 19.7 deep-scrub ok 2024-03-26T04:20:05.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:04 smithi183 ceph-mon[33745]: 19.3 deep-scrub starts 2024-03-26T04:20:05.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:04 smithi183 ceph-mon[33745]: 19.3 deep-scrub ok 2024-03-26T04:20:05.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:04 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:20:05.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:04 smithi183 ceph-mon[33745]: 19.5 deep-scrub starts 2024-03-26T04:20:05.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:04 smithi183 ceph-mon[33745]: 19.5 deep-scrub ok 2024-03-26T04:20:05.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:04 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/806805394' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi016-74313-22"}]: dispatch 2024-03-26T04:20:05.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:04 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1385062143' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi016-74670-7"}]: dispatch 2024-03-26T04:20:05.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:04 smithi183 ceph-mon[33745]: from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi016-74362-10"}]: dispatch 2024-03-26T04:20:05.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:04 smithi183 ceph-mon[33745]: from='client.35222 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-03-26T04:20:05.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:04 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3019850196' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi016-74362-10"}]: dispatch 2024-03-26T04:20:05.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:04 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/883729375' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-26T04:20:05.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:04 smithi183 ceph-mon[33745]: twoxx 2024-03-26T04:20:05.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:04 smithi183 ceph-mon[33745]: from='client.35222 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-03-26T04:20:05.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:04 smithi183 ceph-mon[33745]: from='client.25294 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi016-74295-16"}]: dispatch 2024-03-26T04:20:05.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:04 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3172019969' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi016-74295-16"}]: dispatch 2024-03-26T04:20:05.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:04 smithi183 ceph-mon[33745]: from='client.25306 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi016-74367-10"}]: dispatch 2024-03-26T04:20:05.044 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:04 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1871982092' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi016-74367-10"}]: dispatch 2024-03-26T04:20:05.044 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:04 smithi183 ceph-mon[33745]: from='client.25270 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi016-74692-7"}]: dispatch 2024-03-26T04:20:05.044 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:04 smithi183 ceph-mon[33745]: from='client.24901 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi016-75488-1"}]: dispatch 2024-03-26T04:20:05.044 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:04 smithi183 ceph-mon[33745]: from='client.25210 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi016-74285-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:05.044 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:04 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2267327916' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi016-74692-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:05.044 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:04 smithi183 ceph-mon[33745]: from='client.35192 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:05.044 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:04 smithi183 ceph-mon[33745]: from='client.24901 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi016-75488-1"}]': finished 2024-03-26T04:20:05.044 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:04 smithi183 ceph-mon[33745]: from='client.25210 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi016-74285-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:05.044 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:04 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3950679362' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi016-74781-1"}]: dispatch 2024-03-26T04:20:05.044 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:04 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/287729995' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi016-74508-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:05.044 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:04 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2219155529' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi016-75529-1"}]: dispatch 2024-03-26T04:20:05.044 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:04 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1973575539' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:05.044 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:04 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3740118015' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi016-75488-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:05.045 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:04 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:05.045 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:04 smithi183 ceph-mon[33745]: osdmap e81: 8 total, 8 up, 8 in 2024-03-26T04:20:05.045 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:04 smithi183 ceph-mon[33745]: Health check update: 16 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:20:05.045 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:04 smithi183 ceph-mon[33745]: from='client.34742 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi016-74781-1"}]: dispatch 2024-03-26T04:20:05.045 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:04 smithi183 ceph-mon[33745]: from='client.35246 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi016-74508-1","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:05.045 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:04 smithi183 ceph-mon[33745]: from='client.35264 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi016-75488-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:06.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:05 smithi183 ceph-mon[33745]: 19.2 deep-scrub starts 2024-03-26T04:20:06.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:05 smithi183 ceph-mon[33745]: 19.2 deep-scrub ok 2024-03-26T04:20:06.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:05 smithi183 ceph-mon[33745]: pgmap v79: 756 pgs: 90 creating+peering, 489 active+clean, 177 unknown; 145 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 547 op/s 2024-03-26T04:20:06.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:05 smithi183 ceph-mon[33745]: from='client.35222 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-26T04:20:06.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:05 smithi183 ceph-mon[33745]: from='client.25288 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi016-74313-22"}]: dispatch 2024-03-26T04:20:06.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:05 smithi183 ceph-mon[33745]: from='client.25282 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi016-74670-7"}]: dispatch 2024-03-26T04:20:06.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:05 smithi183 ceph-mon[33745]: from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi016-74362-10"}]: dispatch 2024-03-26T04:20:06.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:05 smithi183 ceph-mon[33745]: from='client.25294 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi016-74295-16"}]: dispatch 2024-03-26T04:20:06.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:05 smithi183 ceph-mon[33745]: from='client.25306 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi016-74367-10"}]: dispatch 2024-03-26T04:20:06.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:05 smithi183 ceph-mon[33745]: from='client.25270 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi016-74692-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:06.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:05 smithi183 ceph-mon[33745]: from='client.34742 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi016-74781-1"}]': finished 2024-03-26T04:20:06.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:05 smithi183 ceph-mon[33745]: from='client.35246 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi016-74508-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:06.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:05 smithi183 ceph-mon[33745]: from='client.35264 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi016-75488-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:20:06.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:05 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3950679362' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi016-74781-1"}]: dispatch 2024-03-26T04:20:06.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:05 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3740118015' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi016-75488-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi016-75488-2"}]: dispatch 2024-03-26T04:20:06.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:05 smithi183 ceph-mon[33745]: osdmap e82: 8 total, 8 up, 8 in 2024-03-26T04:20:06.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:05 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/806805394' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi016-74313-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:06.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:05 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1385062143' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi016-74670-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:06.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:05 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/40094084' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi016-74264-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:06.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:05 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3172019969' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi016-74295-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:06.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:05 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3019850196' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi016-74362-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:06.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:05 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1871982092' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi016-74367-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:06.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:05 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3772872918' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:06.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:05 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:06.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:05 smithi183 ceph-mon[33745]: from='client.34742 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi016-74781-1"}]: dispatch 2024-03-26T04:20:06.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:05 smithi183 ceph-mon[33745]: from='client.24904 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi016-75529-1"}]: dispatch 2024-03-26T04:20:06.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:05 smithi183 ceph-mon[33745]: from='client.35264 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi016-75488-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi016-75488-2"}]: dispatch 2024-03-26T04:20:06.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:05 smithi183 ceph-mon[33745]: from='client.25081 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:06.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[37215]: 19.2 deep-scrub starts 2024-03-26T04:20:06.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[37215]: 19.2 deep-scrub ok 2024-03-26T04:20:06.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[37215]: pgmap v79: 756 pgs: 90 creating+peering, 489 active+clean, 177 unknown; 145 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 547 op/s 2024-03-26T04:20:06.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[37215]: from='client.35222 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-26T04:20:06.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[37215]: from='client.25288 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi016-74313-22"}]: dispatch 2024-03-26T04:20:06.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[37215]: from='client.25282 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi016-74670-7"}]: dispatch 2024-03-26T04:20:06.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[37215]: from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi016-74362-10"}]: dispatch 2024-03-26T04:20:06.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[37215]: from='client.25294 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi016-74295-16"}]: dispatch 2024-03-26T04:20:06.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[37215]: from='client.25306 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi016-74367-10"}]: dispatch 2024-03-26T04:20:06.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[37215]: from='client.25270 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi016-74692-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:06.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[37215]: from='client.34742 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi016-74781-1"}]': finished 2024-03-26T04:20:06.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[37215]: from='client.35246 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi016-74508-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:06.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[37215]: from='client.35264 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi016-75488-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:20:06.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3950679362' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi016-74781-1"}]: dispatch 2024-03-26T04:20:06.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3740118015' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi016-75488-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi016-75488-2"}]: dispatch 2024-03-26T04:20:06.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[37215]: osdmap e82: 8 total, 8 up, 8 in 2024-03-26T04:20:06.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/806805394' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi016-74313-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:06.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1385062143' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi016-74670-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:06.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/40094084' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi016-74264-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:06.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3172019969' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi016-74295-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:06.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[29101]: 19.2 deep-scrub starts 2024-03-26T04:20:06.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[29101]: 19.2 deep-scrub ok 2024-03-26T04:20:06.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[29101]: pgmap v79: 756 pgs: 90 creating+peering, 489 active+clean, 177 unknown; 145 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 547 op/s 2024-03-26T04:20:06.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[29101]: from='client.35222 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-26T04:20:06.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[29101]: from='client.25288 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi016-74313-22"}]: dispatch 2024-03-26T04:20:06.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[29101]: from='client.25282 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi016-74670-7"}]: dispatch 2024-03-26T04:20:06.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[29101]: from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi016-74362-10"}]: dispatch 2024-03-26T04:20:06.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[29101]: from='client.25294 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi016-74295-16"}]: dispatch 2024-03-26T04:20:06.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[29101]: from='client.25306 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi016-74367-10"}]: dispatch 2024-03-26T04:20:06.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[29101]: from='client.25270 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi016-74692-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:06.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[29101]: from='client.34742 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi016-74781-1"}]': finished 2024-03-26T04:20:06.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[29101]: from='client.35246 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi016-74508-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:06.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[29101]: from='client.35264 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi016-75488-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:20:06.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3950679362' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi016-74781-1"}]: dispatch 2024-03-26T04:20:06.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3740118015' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi016-75488-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi016-75488-2"}]: dispatch 2024-03-26T04:20:06.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[29101]: osdmap e82: 8 total, 8 up, 8 in 2024-03-26T04:20:06.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/806805394' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi016-74313-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:06.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1385062143' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi016-74670-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:06.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/40094084' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi016-74264-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:06.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3172019969' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi016-74295-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:06.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3019850196' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi016-74362-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:06.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1871982092' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi016-74367-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:06.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3772872918' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:06.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:06.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[37215]: from='client.34742 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi016-74781-1"}]: dispatch 2024-03-26T04:20:06.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[37215]: from='client.24904 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi016-75529-1"}]: dispatch 2024-03-26T04:20:06.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[37215]: from='client.35264 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi016-75488-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi016-75488-2"}]: dispatch 2024-03-26T04:20:06.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[37215]: from='client.25081 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:06.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3019850196' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi016-74362-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:06.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1871982092' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi016-74367-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:06.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3772872918' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:06.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:06.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[29101]: from='client.34742 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi016-74781-1"}]: dispatch 2024-03-26T04:20:06.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[29101]: from='client.24904 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi016-75529-1"}]: dispatch 2024-03-26T04:20:06.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[29101]: from='client.35264 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi016-75488-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi016-75488-2"}]: dispatch 2024-03-26T04:20:06.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:05 smithi016 ceph-mon[29101]: from='client.25081 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:06.642 INFO:tasks.workunit.client.0.smithi016.stdout:ool application enable","pool": "PoolQuotaPP_smithi016-74285-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:06.642 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:03.326899+0000 mon.a [INF] from='client.25288 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi016-74313-22"}]: dispatch 2024-03-26T04:20:06.642 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:03.327100+0000 mon.a [INF] from='client.35192 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-2","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:06.642 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:03.327400+0000 mon.a [INF] from='client.25282 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi016-74670-7"}]: dispatch 2024-03-26T04:20:06.642 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:04.638557+0000 mon.a [INF] from='client.35222 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-03-26T04:20:06.643 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:04.900326+0000 mon.a [INF] from='client.25288 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi016-74313-22"}]: dispatch 2024-03-26T04:20:06.643 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:04.900510+0000 mon.a [INF] from='client.25282 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi016-74670-7"}]: dispatch 2024-03-26T04:20:06.643 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:04.900655+0000 mon.a [INF] from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi016-74362-10"}]: dispatch 2024-03-26T04:20:06.643 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:04.900842+0000 mon.a [INF] from='client.25294 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi016-74295-16"}]: dispatch 2024-03-26T04:20:06.643 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:04.901015+0000 mon.a [INF] from='client.25306 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi016-74367-10"}]: dispatch 2024-03-26T04:20:06.643 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:04.901159+0000 mon.a [INF] from='client.25270 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi016-74692-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:06.643 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:04.901370+0000 mon.a [INF] from='client.34742 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi016-74781-1"}]': finished 2024-03-26T04:20:06.643 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:04.901426+0000 mon.a [INF] from='client.35246 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi016-74508-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:06.643 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:04.901492+0000 mon.a [INF] from='client.35264 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi016-75488-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:20:06.643 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:04.906435+0000 mon.c [INF] from='client.? v1:172.21.15.16:0/3950679362' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi016-74781-1"}]: dispatch 2024-03-26T04:20:06.643 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:04.906652+0000 mon.c [INF] from='client.? v1:172.21.15.16:0/3740118015' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi016-75488-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi016-75488-2"}]: dispatch 2024-03-26T04:20:06.643 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:04.910947+0000 mon.b [INF] from='client.? v1:172.21.15.16:0/806805394' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi016-74313-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:06.950 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:04.911622+0000 mon.b [INF] from='client.? v1:172.21.15.16:0/1385062143' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi016-74670-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:06.951 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:04.912585+0000 mon.b [INF] from='client.? v1:172.21.15.16:0/40094084' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi016-74264-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:06.951 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:04.913228+0000 mon.b [INF] from='client.? v1:172.21.15.16:0/3172019969' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi016-74295-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:06.951 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:04.914540+0000 mon.b [INF] from='client.? v1:172.21.15.16:0/3019850196' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi016-74362-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:06.951 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:04.914754+0000 mon.b [INF] from='client.? v1:172.21.15.16:0/1871982092' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi016-74367-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:06.951 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:04.915795+0000 mon.b [INF] from='client.? v1:172.21.15.16:0/3772872918' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:06.951 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:05.203734+0000 mon.a [INF] from='client.34742 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi016-74781-1"}]: dispatch 2024-03-26T04:20:06.951 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:05.204062+0000 mon.a [INF] from='client.24904 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi016-75529-1"}]: dispatch 2024-03-26T04:20:06.951 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:05.204240+0000 mon.a [INF] from='client.35264 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReadOpsmithi016-75488-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi016-75488-2"}]: dispatch 2024-03-26T04:20:06.951 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:05.204554+0000 mon.a [INF] from='client.25081 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:06.951 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:05.947905+0000 mon.a [INF] from='client.25288 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi016-74313-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:06.952 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:05.952323+0000 mon.a [INF] from='client.25282 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi016-74670-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:06.952 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:05.952555+0000 mon.a [INF] from='client.25318 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi016-74264-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:06.952 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:05.952891+0000 mon.a [INF] from='client.25294 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi016-74295-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:06.952 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:05.953091+0000 mon.a [INF] from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi016-74362-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:06.952 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:05.953281+0000 mon.a [INF] from='client.25306 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi016-74367-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:06.952 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:05.953513+0000 mon.a [INF] from='client.25324 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:06.952 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:05.953789+0000 mon.a [INF] from='client.25270 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi016-74692-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:20:06.952 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:05.953861+0000 mon.a [INF] from='client.34742 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi016-74781-1"}]': finished 2024-03-26T04:20:06.952 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:05.953910+0000 mon.a [INF] from='client.24904 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi016-75529-1"}]': finished 2024-03-26T04:20:06.952 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:05.953957+0000 mon.a [INF] from='client.25081 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:06.952 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:05.959253+0000 mon.b [INF] from='client.? v1:172.21.15.16:0/2219155529' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi016-75529-1"}]: dispatch 2024-03-26T04:20:06.952 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:05.959555+0000 mon.b [INF] from='client.? v1:172.21.15.16:0/2267327916' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi016-74692-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi016-74692-7"}]: dispatch 2024-03-26T04:20:06.952 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:05.960468+0000 mon.b [INF] from='client.? v1:172.21.15.16:0/1757055001' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi016-74285-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:06.952 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:05.961169+0000 mon.b [INF] from='client.? v1:172.21.15.16:0/1973575539' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-4", "tierpool": "test-rados-api-smithi016-74534-4-cache"}]: dispatch 2024-03-26T04:20:06.953 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:06.037041+0000 mon.c [INF] from='client.? v1:172.21.15.16:0/287729995' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi016-74508-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:06.953 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:06.232512+0000 mon.c [INF] from='client.? v1:172.21.15.16:0/1369669707' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:06.953 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:06.488917+0000 mon.a [INF] from='client.35246 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi016-74508-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:06.953 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:06.489691+0000 mon.a [INF] from='client.35279 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:06.974 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:06.495613+0000 mon.a [INF] from='client.24904 ' entity='client.admin' cmd=[{" open_pools_parallel: process_1_[75239]: starting. 2024-03-26T04:20:06.974 INFO:tasks.workunit.client.0.smithi016.stdout: open_pools_parallel: process_1_[75239]: creating pool ceph_test_rados_open_pools_parallel.smithi016-75163 2024-03-26T04:20:06.975 INFO:tasks.workunit.client.0.smithi016.stdout: open_pools_parallel: process_1_[75239]: created object 0... 2024-03-26T04:20:06.975 INFO:tasks.workunit.client.0.smithi016.stdout: open_pools_parallel: process_1_[75239]: created object 25... 2024-03-26T04:20:06.975 INFO:tasks.workunit.client.0.smithi016.stdout: open_pools_parallel: process_1_[75239]: created object 49... 2024-03-26T04:20:06.975 INFO:tasks.workunit.client.0.smithi016.stdout: open_pools_parallel: process_1_[75239]: finishing. 2024-03-26T04:20:06.975 INFO:tasks.workunit.client.0.smithi016.stdout: open_pools_parallel: process_1_[75239]: shutting down. 2024-03-26T04:20:06.975 INFO:tasks.workunit.client.0.smithi016.stdout: open_pools_parallel: process_2_[75241]: starting. 2024-03-26T04:20:06.975 INFO:tasks.workunit.client.0.smithi016.stdout: open_pools_parallel: process_2_[75241]: rados_pool_create. 2024-03-26T04:20:06.975 INFO:tasks.workunit.client.0.smithi016.stdout: open_pools_parallel: process_2_[75241]: rados_ioctx_create. 2024-03-26T04:20:06.975 INFO:tasks.workunit.client.0.smithi016.stdout: open_pools_parallel: process_2_[75241]: shutting down. 2024-03-26T04:20:06.975 INFO:tasks.workunit.client.0.smithi016.stdout: open_pools_parallel: ******************************* 2024-03-26T04:20:06.975 INFO:tasks.workunit.client.0.smithi016.stdout: open_pools_parallel: process_3_[76667]: starting. 2024-03-26T04:20:06.975 INFO:tasks.workunit.client.0.smithi016.stdout: open_pools_parallel: process_3_[76667]: creating pool ceph_test_rados_open_pools_parallel.smithi016-75163 2024-03-26T04:20:06.975 INFO:tasks.workunit.client.0.smithi016.stdout: open_pools_parallel: process_3_[76667]: created object 0... 2024-03-26T04:20:06.975 INFO:tasks.workunit.client.0.smithi016.stdout: open_pools_parallel: process_3_[76667]: created object 25... 2024-03-26T04:20:06.976 INFO:tasks.workunit.client.0.smithi016.stdout: open_pools_parallel: process_3_[76667]: created object 49... 2024-03-26T04:20:06.976 INFO:tasks.workunit.client.0.smithi016.stdout: open_pools_parallel: process_3_[76667]: finishing. 2024-03-26T04:20:06.976 INFO:tasks.workunit.client.0.smithi016.stdout: open_pools_parallel: process_3_[76667]: shutting down. 2024-03-26T04:20:06.976 INFO:tasks.workunit.client.0.smithi016.stdout: open_pools_parallel: ******************************* 2024-03-26T04:20:06.976 INFO:tasks.workunit.client.0.smithi016.stdout: open_pools_parallel: process_4_[76668]: starting. 2024-03-26T04:20:06.976 INFO:tasks.workunit.client.0.smithi016.stdout: open_pools_parallel: process_4_[76668]: rados_pool_create. 2024-03-26T04:20:06.976 INFO:tasks.workunit.client.0.smithi016.stdout: open_pools_parallel: process_4_[76668]: rados_ioctx_create. 2024-03-26T04:20:06.976 INFO:tasks.workunit.client.0.smithi016.stdout: open_pools_parallel: process_4_[76668]: shutting down. 2024-03-26T04:20:06.976 INFO:tasks.workunit.client.0.smithi016.stdout: open_pools_parallel: ******************************* 2024-03-26T04:20:06.976 INFO:tasks.workunit.client.0.smithi016.stdout: open_pools_parallel: ******************************* 2024-03-26T04:20:06.976 INFO:tasks.workunit.client.0.smithi016.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-03-26T04:20:07.262 INFO:tasks.workunit.client.0.smithi016.stdout: delete_pools_parallel: process_1_[75261]: starting. 2024-03-26T04:20:07.262 INFO:tasks.workunit.client.0.smithi016.stdout: delete_pools_parallel: process_1_[75261]: creating pool ceph_test_rados_delete_pools_parallel.smithi016-75217 2024-03-26T04:20:07.262 INFO:tasks.workunit.client.0.smithi016.stdout: delete_pools_parallel: process_1_[75261]: created object 0... 2024-03-26T04:20:07.262 INFO:tasks.workunit.client.0.smithi016.stdout: delete_pools_parallel: process_1_[75261]: created object 25... 2024-03-26T04:20:07.262 INFO:tasks.workunit.client.0.smithi016.stdout: delete_pools_parallel: process_1_[75261]: created object 49... 2024-03-26T04:20:07.262 INFO:tasks.workunit.client.0.smithi016.stdout: delete_pools_parallel: process_1_[75261]: finishing. 2024-03-26T04:20:07.262 INFO:tasks.workunit.client.0.smithi016.stdout: delete_pools_parallel: process_1_[75261]: shutting down. 2024-03-26T04:20:07.263 INFO:tasks.workunit.client.0.smithi016.stdout: delete_pools_parallel: process_2_[75266]: starting. 2024-03-26T04:20:07.263 INFO:tasks.workunit.client.0.smithi016.stdout: delete_pools_parallel: process_2_[75266]: deleting pool ceph_test_rados_delete_pools_parallel.smithi016-75217 2024-03-26T04:20:07.263 INFO:tasks.workunit.client.0.smithi016.stdout: delete_pools_parallel: process_2_[75266]: shutting down. 2024-03-26T04:20:07.263 INFO:tasks.workunit.client.0.smithi016.stdout: delete_pools_parallel: ******************************* 2024-03-26T04:20:07.263 INFO:tasks.workunit.client.0.smithi016.stdout: delete_pools_parallel: process_3_[76664]: starting. 2024-03-26T04:20:07.263 INFO:tasks.workunit.client.0.smithi016.stdout: delete_pools_parallel: process_3_[76664]: creating pool ceph_test_rados_delete_pools_parallel.smithi016-75217 2024-03-26T04:20:07.263 INFO:tasks.workunit.client.0.smithi016.stdout: delete_pools_parallel: process_3_[76664]: created object 0... 2024-03-26T04:20:07.263 INFO:tasks.workunit.client.0.smithi016.stdout: delete_pools_parallel: process_3_[76664]: created object 25... 2024-03-26T04:20:07.263 INFO:tasks.workunit.client.0.smithi016.stdout: delete_pools_parallel: process_3_[76664]: created object 49... 2024-03-26T04:20:07.263 INFO:tasks.workunit.client.0.smithi016.stdout: delete_pools_parallel: process_3_[76664]: finishing. 2024-03-26T04:20:07.263 INFO:tasks.workunit.client.0.smithi016.stdout: delete_pools_parallel: process_3_[76664]: shutting down. 2024-03-26T04:20:07.264 INFO:tasks.workunit.client.0.smithi016.stdout: delete_pools_parallel: ******************************* 2024-03-26T04:20:07.264 INFO:tasks.workunit.client.0.smithi016.stdout: delete_pools_parallel: process_5_[76666]: starting. 2024-03-26T04:20:07.264 INFO:tasks.workunit.client.0.smithi016.stdout: delete_pools_parallel: process_5_[76666]: listing objects. 2024-03-26T04:20:07.264 INFO:tasks.workunit.client.0.smithi016.stdout: delete_pools_parallel: process_5_[76666]: listed object 0... 2024-03-26T04:20:07.264 INFO:tasks.workunit.client.0.smithi016.stdout: delete_pools_parallel: process_5_[76666]: listed object 25... 2024-03-26T04:20:07.264 INFO:tasks.workunit.client.0.smithi016.stdout: delete_pools_parallel: process_5_[76666]: saw 50 objects 2024-03-26T04:20:07.264 INFO:tasks.workunit.client.0.smithi016.stdout: delete_pools_parallel: process_5_[76666]: shutting down. 2024-03-26T04:20:07.264 INFO:tasks.workunit.client.0.smithi016.stdout: delete_pools_parallel: ******************************* 2024-03-26T04:20:07.264 INFO:tasks.workunit.client.0.smithi016.stdout: delete_pools_parallel: process_4_[76665]: starting. 2024-03-26T04:20:07.264 INFO:tasks.workunit.client.0.smithi016.stdout: delete_pools_parallel: process_4_[76665]: deleting pool ceph_test_rados_delete_pools_parallel.smithi016-75217 2024-03-26T04:20:07.264 INFO:tasks.workunit.client.0.smithi016.stdout: delete_pools_parallel: process_4_[76665]: shutting down. 2024-03-26T04:20:07.264 INFO:tasks.workunit.client.0.smithi016.stdout: delete_pools_parallel: ******************************* 2024-03-26T04:20:07.264 INFO:tasks.workunit.client.0.smithi016.stdout: delete_pools_parallel: ******************************* 2024-03-26T04:20:07.264 INFO:tasks.workunit.client.0.smithi016.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-03-26T04:20:07.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:06 smithi183 ceph-mon[33745]: pgmap v82: 836 pgs: 13 creating+peering, 452 active+clean, 371 unknown; 145 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 547 op/s 2024-03-26T04:20:07.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:06 smithi183 ceph-mon[33745]: from='client.25288 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi016-74313-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:07.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:06 smithi183 ceph-mon[33745]: from='client.25282 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi016-74670-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:07.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:06 smithi183 ceph-mon[33745]: from='client.25318 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi016-74264-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:07.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:06 smithi183 ceph-mon[33745]: from='client.25294 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi016-74295-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:07.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:06 smithi183 ceph-mon[33745]: from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi016-74362-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:07.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:06 smithi183 ceph-mon[33745]: from='client.25306 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi016-74367-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:07.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:06 smithi183 ceph-mon[33745]: from='client.25324 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:07.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:06 smithi183 ceph-mon[33745]: from='client.25270 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi016-74692-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:20:07.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:06 smithi183 ceph-mon[33745]: from='client.34742 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi016-74781-1"}]': finished 2024-03-26T04:20:07.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:06 smithi183 ceph-mon[33745]: from='client.24904 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi016-75529-1"}]': finished 2024-03-26T04:20:07.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:06 smithi183 ceph-mon[33745]: from='client.25081 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:07.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:06 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2219155529' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi016-75529-1"}]: dispatch 2024-03-26T04:20:07.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:06 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2267327916' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi016-74692-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi016-74692-7"}]: dispatch 2024-03-26T04:20:07.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:06 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1757055001' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi016-74285-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:07.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:06 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1973575539' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-4", "tierpool": "test-rados-api-smithi016-74534-4-cache"}]: dispatch 2024-03-26T04:20:07.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:06 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/287729995' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-03-26T04:20:07.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:06 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/287729995' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi016-74508-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:07.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:06 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:07.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:06 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1369669707' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:07.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:06 smithi183 ceph-mon[33745]: osdmap e83: 8 total, 8 up, 8 in 2024-03-26T04:20:07.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:06 smithi183 ceph-mon[33745]: from='client.35246 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi016-74508-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:07.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:06 smithi183 ceph-mon[33745]: from='client.35279 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:07.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:06 smithi183 ceph-mon[33745]: from='client.24904 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi016-75529-1"}]: dispatch 2024-03-26T04:20:07.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:06 smithi183 ceph-mon[33745]: from='client.25270 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi016-74692-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi016-74692-7"}]: dispatch 2024-03-26T04:20:07.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:06 smithi183 ceph-mon[33745]: from='client.25210 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi016-74285-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:07.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:06 smithi183 ceph-mon[33745]: from='client.25081 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-4", "tierpool": "test-rados-api-smithi016-74534-4-cache"}]: dispatch 2024-03-26T04:20:07.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:06 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:20:07.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:06 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:20:07.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:06 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/883729375' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-26T04:20:07.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:06 smithi183 ceph-mon[33745]: threexx 2024-03-26T04:20:07.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:06 smithi183 ceph-mon[33745]: from='client.35222 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-26T04:20:07.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[37215]: pgmap v82: 836 pgs: 13 creating+peering, 452 active+clean, 371 unknown; 145 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 547 op/s 2024-03-26T04:20:07.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[37215]: from='client.25288 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi016-74313-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:07.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[37215]: from='client.25282 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi016-74670-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:07.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[37215]: from='client.25318 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi016-74264-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:07.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[37215]: from='client.25294 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi016-74295-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:07.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[37215]: from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi016-74362-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:07.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[37215]: from='client.25306 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi016-74367-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:07.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[37215]: from='client.25324 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:07.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[37215]: from='client.25270 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi016-74692-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:20:07.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[37215]: from='client.34742 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi016-74781-1"}]': finished 2024-03-26T04:20:07.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[37215]: from='client.24904 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi016-75529-1"}]': finished 2024-03-26T04:20:07.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[37215]: from='client.25081 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:07.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2219155529' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi016-75529-1"}]: dispatch 2024-03-26T04:20:07.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2267327916' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi016-74692-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi016-74692-7"}]: dispatch 2024-03-26T04:20:07.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1757055001' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi016-74285-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:07.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1973575539' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-4", "tierpool": "test-rados-api-smithi016-74534-4-cache"}]: dispatch 2024-03-26T04:20:07.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/287729995' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-03-26T04:20:07.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[29101]: pgmap v82: 836 pgs: 13 creating+peering, 452 active+clean, 371 unknown; 145 MiB data, 712 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 37 MiB/s wr, 547 op/s 2024-03-26T04:20:07.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[29101]: from='client.25288 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi016-74313-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:07.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[29101]: from='client.25282 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi016-74670-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:07.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[29101]: from='client.25318 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi016-74264-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:07.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[29101]: from='client.25294 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi016-74295-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:07.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[29101]: from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi016-74362-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:07.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[29101]: from='client.25306 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi016-74367-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:07.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[29101]: from='client.25324 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:07.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[29101]: from='client.25270 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi016-74692-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:20:07.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[29101]: from='client.34742 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi016-74781-1"}]': finished 2024-03-26T04:20:07.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[29101]: from='client.24904 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi016-75529-1"}]': finished 2024-03-26T04:20:07.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[29101]: from='client.25081 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:07.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2219155529' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi016-75529-1"}]: dispatch 2024-03-26T04:20:07.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2267327916' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi016-74692-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi016-74692-7"}]: dispatch 2024-03-26T04:20:07.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1757055001' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi016-74285-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:07.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1973575539' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-4", "tierpool": "test-rados-api-smithi016-74534-4-cache"}]: dispatch 2024-03-26T04:20:07.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/287729995' entity='client.admin' cmd=[{"prefix":"osd dump"}]: dispatch 2024-03-26T04:20:07.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/287729995' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi016-74508-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:07.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:07.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1369669707' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:07.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[37215]: osdmap e83: 8 total, 8 up, 8 in 2024-03-26T04:20:07.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[37215]: from='client.35246 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi016-74508-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:07.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[37215]: from='client.35279 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:07.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[37215]: from='client.24904 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi016-75529-1"}]: dispatch 2024-03-26T04:20:07.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[37215]: from='client.25270 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi016-74692-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi016-74692-7"}]: dispatch 2024-03-26T04:20:07.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[37215]: from='client.25210 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi016-74285-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:07.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[37215]: from='client.25081 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-4", "tierpool": "test-rados-api-smithi016-74534-4-cache"}]: dispatch 2024-03-26T04:20:07.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:20:07.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:20:07.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/883729375' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-26T04:20:07.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[37215]: threexx 2024-03-26T04:20:07.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[37215]: from='client.35222 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-26T04:20:07.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/287729995' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi016-74508-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:07.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:07.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1369669707' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:07.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[29101]: osdmap e83: 8 total, 8 up, 8 in 2024-03-26T04:20:07.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[29101]: from='client.35246 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi016-74508-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:07.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[29101]: from='client.35279 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:07.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[29101]: from='client.24904 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi016-75529-1"}]: dispatch 2024-03-26T04:20:07.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[29101]: from='client.25270 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi016-74692-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi016-74692-7"}]: dispatch 2024-03-26T04:20:07.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[29101]: from='client.25210 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi016-74285-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:07.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[29101]: from='client.25081 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-4", "tierpool": "test-rados-api-smithi016-74534-4-cache"}]: dispatch 2024-03-26T04:20:07.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:20:07.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:20:07.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/883729375' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-26T04:20:07.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[29101]: threexx 2024-03-26T04:20:07.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:06 smithi016 ceph-mon[29101]: from='client.35222 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-26T04:20:08.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:08 smithi183 ceph-mon[33745]: from='client.35222 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-03-26T04:20:08.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:08 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/883729375' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-26T04:20:08.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:08 smithi183 ceph-mon[33745]: fourxx 2024-03-26T04:20:08.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:08 smithi183 ceph-mon[33745]: 19.8 deep-scrub starts 2024-03-26T04:20:08.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:08 smithi183 ceph-mon[33745]: from='client.35222 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-26T04:20:08.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:08 smithi183 ceph-mon[33745]: 19.8 deep-scrub ok 2024-03-26T04:20:08.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:08 smithi183 ceph-mon[33745]: from='client.35264 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi016-75488-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi016-75488-2"}]': finished 2024-03-26T04:20:08.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:08 smithi183 ceph-mon[33745]: from='client.25288 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi016-74313-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:20:08.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:08 smithi183 ceph-mon[33745]: from='client.25282 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi016-74670-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:20:08.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:08 smithi183 ceph-mon[33745]: from='client.25318 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi016-74264-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:08.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:08 smithi183 ceph-mon[33745]: from='client.25294 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi016-74295-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:20:08.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:08 smithi183 ceph-mon[33745]: from='client.25291 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi016-74362-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:20:08.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:08 smithi183 ceph-mon[33745]: from='client.25306 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi016-74367-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:20:08.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:08 smithi183 ceph-mon[33745]: from='client.25324 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:08.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:08 smithi183 ceph-mon[33745]: from='client.35246 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi016-74508-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:08.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:08 smithi183 ceph-mon[33745]: from='client.35279 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:08.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:08 smithi183 ceph-mon[33745]: from='client.24904 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi016-75529-1"}]': finished 2024-03-26T04:20:08.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:08 smithi183 ceph-mon[33745]: from='client.25210 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi016-74285-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:08.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:08 smithi183 ceph-mon[33745]: from='client.25081 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-4", "tierpool": "test-rados-api-smithi016-74534-4-cache"}]': finished 2024-03-26T04:20:08.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:08 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1973575539' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-4", "overlaypool": "test-rados-api-smithi016-74534-4-cache"}]: dispatch 2024-03-26T04:20:08.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:08 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1385062143' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi016-74670-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi016-74670-7"}]: dispatch 2024-03-26T04:20:08.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:08 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/806805394' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi016-74313-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi016-74313-22"}]: dispatch 2024-03-26T04:20:08.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:08 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3019850196' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi016-74362-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi016-74362-10"}]: dispatch 2024-03-26T04:20:08.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:08 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3172019969' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi016-74295-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi016-74295-16"}]: dispatch 2024-03-26T04:20:08.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:08 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3878746505' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi016-74781-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:08.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:08 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1871982092' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi016-74367-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi016-74367-10"}]: dispatch 2024-03-26T04:20:08.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:08 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1757055001' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi016-74285-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-26T04:20:08.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:08 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/287729995' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi016-74508-1","app": "app2"}]: dispatch 2024-03-26T04:20:08.295 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:08 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/287729995' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi016-74508-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:08.295 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:08 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/929019900' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi016-75529-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:08.295 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:08 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:08.295 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:08 smithi183 ceph-mon[33745]: osdmap e84: 8 total, 8 up, 8 in 2024-03-26T04:20:08.295 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:08 smithi183 ceph-mon[33745]: from='client.25081 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-4", "overlaypool": "test-rados-api-smithi016-74534-4-cache"}]: dispatch 2024-03-26T04:20:08.295 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:08 smithi183 ceph-mon[33745]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi016-74781-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:08.295 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:08 smithi183 ceph-mon[33745]: from='client.25282 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi016-74670-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi016-74670-7"}]: dispatch 2024-03-26T04:20:08.295 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:08 smithi183 ceph-mon[33745]: from='client.25288 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi016-74313-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi016-74313-22"}]: dispatch 2024-03-26T04:20:08.295 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:08 smithi183 ceph-mon[33745]: from='client.35246 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi016-74508-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:08.295 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:08 smithi183 ceph-mon[33745]: from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi016-74362-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi016-74362-10"}]: dispatch 2024-03-26T04:20:08.295 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:08 smithi183 ceph-mon[33745]: from='client.35309 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi016-75529-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:08.295 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:08 smithi183 ceph-mon[33745]: from='client.25294 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi016-74295-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi016-74295-16"}]: dispatch 2024-03-26T04:20:08.295 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:08 smithi183 ceph-mon[33745]: from='client.25306 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi016-74367-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi016-74367-10"}]: dispatch 2024-03-26T04:20:08.295 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:08 smithi183 ceph-mon[33745]: from='client.25210 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi016-74285-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-26T04:20:08.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[37215]: from='client.35222 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-03-26T04:20:08.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/883729375' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-26T04:20:08.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[37215]: fourxx 2024-03-26T04:20:08.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[37215]: 19.8 deep-scrub starts 2024-03-26T04:20:08.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[37215]: from='client.35222 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-26T04:20:08.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[37215]: 19.8 deep-scrub ok 2024-03-26T04:20:08.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[37215]: from='client.35264 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi016-75488-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi016-75488-2"}]': finished 2024-03-26T04:20:08.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[37215]: from='client.25288 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi016-74313-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:20:08.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[37215]: from='client.25282 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi016-74670-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:20:08.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[37215]: from='client.25318 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi016-74264-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:08.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[37215]: from='client.25294 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi016-74295-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:20:08.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[37215]: from='client.25291 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi016-74362-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:20:08.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[37215]: from='client.25306 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi016-74367-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:20:08.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[37215]: from='client.25324 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:08.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[37215]: from='client.35246 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi016-74508-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:08.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[37215]: from='client.35279 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:08.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[37215]: from='client.24904 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi016-75529-1"}]': finished 2024-03-26T04:20:08.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[37215]: from='client.25210 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi016-74285-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:08.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[37215]: from='client.25081 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-4", "tierpool": "test-rados-api-smithi016-74534-4-cache"}]': finished 2024-03-26T04:20:08.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1973575539' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-4", "overlaypool": "test-rados-api-smithi016-74534-4-cache"}]: dispatch 2024-03-26T04:20:08.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[29101]: from='client.35222 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["threexx"]}]': finished 2024-03-26T04:20:08.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/883729375' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-26T04:20:08.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[29101]: fourxx 2024-03-26T04:20:08.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[29101]: 19.8 deep-scrub starts 2024-03-26T04:20:08.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[29101]: from='client.35222 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["fourxx"]}]: dispatch 2024-03-26T04:20:08.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[29101]: 19.8 deep-scrub ok 2024-03-26T04:20:08.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[29101]: from='client.35264 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReadOpsmithi016-75488-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi016-75488-2"}]': finished 2024-03-26T04:20:08.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[29101]: from='client.25288 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi016-74313-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:20:08.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[29101]: from='client.25282 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi016-74670-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:20:08.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[29101]: from='client.25318 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi016-74264-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:08.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[29101]: from='client.25294 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi016-74295-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:20:08.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[29101]: from='client.25291 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi016-74362-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:20:08.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[29101]: from='client.25306 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi016-74367-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:20:08.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[29101]: from='client.25324 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:08.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[29101]: from='client.35246 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi016-74508-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:08.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[29101]: from='client.35279 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:08.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[29101]: from='client.24904 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi016-75529-1"}]': finished 2024-03-26T04:20:08.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[29101]: from='client.25210 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi016-74285-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:08.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[29101]: from='client.25081 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-4", "tierpool": "test-rados-api-smithi016-74534-4-cache"}]': finished 2024-03-26T04:20:08.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1973575539' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-4", "overlaypool": "test-rados-api-smithi016-74534-4-cache"}]: dispatch 2024-03-26T04:20:08.418 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1385062143' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi016-74670-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi016-74670-7"}]: dispatch 2024-03-26T04:20:08.418 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/806805394' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi016-74313-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi016-74313-22"}]: dispatch 2024-03-26T04:20:08.418 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3019850196' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi016-74362-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi016-74362-10"}]: dispatch 2024-03-26T04:20:08.418 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3172019969' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi016-74295-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi016-74295-16"}]: dispatch 2024-03-26T04:20:08.418 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3878746505' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi016-74781-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:08.418 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1871982092' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi016-74367-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi016-74367-10"}]: dispatch 2024-03-26T04:20:08.419 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1757055001' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi016-74285-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-26T04:20:08.419 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/287729995' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi016-74508-1","app": "app2"}]: dispatch 2024-03-26T04:20:08.419 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/287729995' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi016-74508-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:08.419 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/929019900' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi016-75529-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:08.419 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:08.419 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[37215]: osdmap e84: 8 total, 8 up, 8 in 2024-03-26T04:20:08.419 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[37215]: from='client.25081 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-4", "overlaypool": "test-rados-api-smithi016-74534-4-cache"}]: dispatch 2024-03-26T04:20:08.419 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[37215]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi016-74781-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:08.419 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[37215]: from='client.25282 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi016-74670-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi016-74670-7"}]: dispatch 2024-03-26T04:20:08.419 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1385062143' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi016-74670-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi016-74670-7"}]: dispatch 2024-03-26T04:20:08.419 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/806805394' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi016-74313-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi016-74313-22"}]: dispatch 2024-03-26T04:20:08.420 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3019850196' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi016-74362-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi016-74362-10"}]: dispatch 2024-03-26T04:20:08.420 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3172019969' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi016-74295-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi016-74295-16"}]: dispatch 2024-03-26T04:20:08.420 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3878746505' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi016-74781-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:08.420 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1871982092' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi016-74367-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi016-74367-10"}]: dispatch 2024-03-26T04:20:08.420 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1757055001' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi016-74285-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-26T04:20:08.420 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/287729995' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi016-74508-1","app": "app2"}]: dispatch 2024-03-26T04:20:08.420 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/287729995' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi016-74508-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:08.420 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/929019900' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi016-75529-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:08.420 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:08.420 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[29101]: osdmap e84: 8 total, 8 up, 8 in 2024-03-26T04:20:08.420 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[29101]: from='client.25081 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-4", "overlaypool": "test-rados-api-smithi016-74534-4-cache"}]: dispatch 2024-03-26T04:20:08.420 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[29101]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi016-74781-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:08.420 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[29101]: from='client.25282 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi016-74670-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi016-74670-7"}]: dispatch 2024-03-26T04:20:08.421 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[37215]: from='client.25288 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi016-74313-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi016-74313-22"}]: dispatch 2024-03-26T04:20:08.421 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[37215]: from='client.35246 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi016-74508-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:08.421 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[37215]: from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi016-74362-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi016-74362-10"}]: dispatch 2024-03-26T04:20:08.421 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[37215]: from='client.35309 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi016-75529-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:08.421 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[37215]: from='client.25294 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi016-74295-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi016-74295-16"}]: dispatch 2024-03-26T04:20:08.421 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[37215]: from='client.25306 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi016-74367-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi016-74367-10"}]: dispatch 2024-03-26T04:20:08.422 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[37215]: from='client.25210 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi016-74285-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-26T04:20:08.422 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[29101]: from='client.25288 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi016-74313-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi016-74313-22"}]: dispatch 2024-03-26T04:20:08.422 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[29101]: from='client.35246 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi016-74508-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:08.422 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[29101]: from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi016-74362-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi016-74362-10"}]: dispatch 2024-03-26T04:20:08.422 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[29101]: from='client.35309 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi016-75529-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:08.422 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[29101]: from='client.25294 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi016-74295-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi016-74295-16"}]: dispatch 2024-03-26T04:20:08.422 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[29101]: from='client.25306 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi016-74367-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi016-74367-10"}]: dispatch 2024-03-26T04:20:08.422 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-mon[29101]: from='client.25210 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi016-74285-3", "field": "max_bytes", "val": "4096"}]: dispatch 2024-03-26T04:20:09.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:20:08 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:20:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:20:09.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:09 smithi016 ceph-mon[37215]: from='client.35222 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-03-26T04:20:09.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:09 smithi016 ceph-mon[37215]: from='client.25270 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi016-74692-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi016-74692-7"}]': finished 2024-03-26T04:20:09.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:09 smithi016 ceph-mon[37215]: from='client.25081 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-4", "overlaypool": "test-rados-api-smithi016-74534-4-cache"}]': finished 2024-03-26T04:20:09.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:09 smithi016 ceph-mon[37215]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi016-74781-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:09.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:09 smithi016 ceph-mon[37215]: from='client.35246 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi016-74508-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:09.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:09 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/929019900' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi016-75529-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi016-75529-2"}]: dispatch 2024-03-26T04:20:09.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:09 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1973575539' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-4-cache", "mode": "writeback"}]: dispatch 2024-03-26T04:20:09.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:09 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/287729995' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi016-74508-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-03-26T04:20:09.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:09 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:09.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:09 smithi016 ceph-mon[29101]: from='client.35222 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-03-26T04:20:09.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:09 smithi016 ceph-mon[29101]: from='client.25270 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi016-74692-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi016-74692-7"}]': finished 2024-03-26T04:20:09.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:09 smithi016 ceph-mon[29101]: from='client.25081 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-4", "overlaypool": "test-rados-api-smithi016-74534-4-cache"}]': finished 2024-03-26T04:20:09.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:09 smithi016 ceph-mon[29101]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi016-74781-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:09.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:09 smithi016 ceph-mon[29101]: from='client.35246 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi016-74508-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:09.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:09 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/929019900' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi016-75529-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi016-75529-2"}]: dispatch 2024-03-26T04:20:09.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:09 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1973575539' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-4-cache", "mode": "writeback"}]: dispatch 2024-03-26T04:20:09.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:09 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/287729995' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi016-74508-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-03-26T04:20:09.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:09 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:09.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:09 smithi183 ceph-mon[33745]: from='client.35222 ' entity='client.admin' cmd='[{"prefix":"log", "logtext":["fourxx"]}]': finished 2024-03-26T04:20:09.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:09 smithi183 ceph-mon[33745]: from='client.25270 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi016-74692-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi016-74692-7"}]': finished 2024-03-26T04:20:09.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:09 smithi183 ceph-mon[33745]: from='client.25081 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-4", "overlaypool": "test-rados-api-smithi016-74534-4-cache"}]': finished 2024-03-26T04:20:09.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:09 smithi183 ceph-mon[33745]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi016-74781-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:09.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:09 smithi183 ceph-mon[33745]: from='client.35246 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMisc_smithi016-74508-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:09.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:09 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/929019900' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi016-75529-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi016-75529-2"}]: dispatch 2024-03-26T04:20:09.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:09 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1973575539' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-4-cache", "mode": "writeback"}]: dispatch 2024-03-26T04:20:09.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:09 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/287729995' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi016-74508-1","app":"dne","key":"key","value":"value"}]: dispatch 2024-03-26T04:20:09.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:09 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:10.015 INFO:tasks.workunit.client.0.smithi016.stdout:prefix": "osd crush rule rm", "name":"ListObjectssmithi016-75529-1"}]: dispatch 2024-03-26T04:20:10.015 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:06.495854+0000 mon.a [INF] from='client.25270 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi016-74692-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi016-74692-7"}]: dispatch 2024-03-26T04:20:10.016 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:06.496140+0000 mon.a [INF] from='client.25210 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi016-74285-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:10.016 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:06.496394+0000 mon.a [INF] from='client.25081 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-4", "tierpool": "test-rados-api-smithi016-74534-4-cache"}]: dispatch 2024-03-26T04:20:10.016 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:06.503588+0000 mon.a [INF] from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:20:10.016 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:06.643005+0000 mon.c [INF] from='client.? v1:172.21.15.16:0/883729375' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-26T04:20:10.016 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:06.643145+0000 client.admin [INF] threexx 2024-03-26T04:20:10.016 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: got: 2024-03-26T04:20:06.643308+0000 mon.a [INF] from='client.35222 ' entity='client.admin' cmd=[{"prefix":"log", "logtext":["threexx"]}]: dispatch 2024-03-26T04:20:10.016 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (8320 ms) 2024-03-26T04:20:10.016 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (21290 ms total) 2024-03-26T04:20:10.016 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: 2024-03-26T04:20:10.016 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: [----------] Global test environment tear-down 2024-03-26T04:20:10.016 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (21290 ms total) 2024-03-26T04:20:10.016 INFO:tasks.workunit.client.0.smithi016.stdout: api_cmd: [ PASSED ] 4 tests. 2024-03-26T04:20:10.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:10 smithi016 ceph-mon[37215]: 19.4 deep-scrub starts 2024-03-26T04:20:10.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:10 smithi016 ceph-mon[37215]: 19.4 deep-scrub ok 2024-03-26T04:20:10.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:10 smithi016 ceph-mon[37215]: from='client.35309 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi016-75529-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:20:10.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:10 smithi016 ceph-mon[37215]: from='client.25210 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi016-74285-3", "field": "max_bytes", "val": "4096"}]': finished 2024-03-26T04:20:10.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:10 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/287729995' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi016-74508-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T04:20:10.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:10 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3878746505' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi016-74781-6", "pg_num": 4}]: dispatch 2024-03-26T04:20:10.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:10 smithi016 ceph-mon[37215]: 19.1 deep-scrub starts 2024-03-26T04:20:10.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:10 smithi016 ceph-mon[37215]: 19.1 deep-scrub ok 2024-03-26T04:20:10.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:10 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:10.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:10 smithi016 ceph-mon[37215]: osdmap e85: 8 total, 8 up, 8 in 2024-03-26T04:20:10.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:10 smithi016 ceph-mon[37215]: from='client.35309 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi016-75529-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi016-75529-2"}]: dispatch 2024-03-26T04:20:10.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:10 smithi016 ceph-mon[37215]: from='client.35246 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi016-74508-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T04:20:10.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:10 smithi016 ceph-mon[37215]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi016-74781-6", "pg_num": 4}]: dispatch 2024-03-26T04:20:10.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:10 smithi016 ceph-mon[37215]: from='client.25282 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi016-74670-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi016-74670-7"}]': finished 2024-03-26T04:20:10.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:10 smithi016 ceph-mon[37215]: from='client.25288 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi016-74313-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi016-74313-22"}]': finished 2024-03-26T04:20:10.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:10 smithi016 ceph-mon[37215]: from='client.25291 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi016-74362-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi016-74362-10"}]': finished 2024-03-26T04:20:10.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:10 smithi016 ceph-mon[37215]: from='client.25294 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi016-74295-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi016-74295-16"}]': finished 2024-03-26T04:20:10.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:10 smithi016 ceph-mon[37215]: from='client.25306 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi016-74367-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi016-74367-10"}]': finished 2024-03-26T04:20:10.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:10 smithi016 ceph-mon[37215]: osdmap e86: 8 total, 8 up, 8 in 2024-03-26T04:20:10.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:10 smithi016 ceph-mon[37215]: Health check update: 9 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:20:10.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:10 smithi016 ceph-mon[29101]: 19.4 deep-scrub starts 2024-03-26T04:20:10.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:10 smithi016 ceph-mon[29101]: 19.4 deep-scrub ok 2024-03-26T04:20:10.917 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:10 smithi016 ceph-mon[29101]: from='client.35309 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi016-75529-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:20:10.917 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:10 smithi016 ceph-mon[29101]: from='client.25210 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi016-74285-3", "field": "max_bytes", "val": "4096"}]': finished 2024-03-26T04:20:10.917 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:10 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/287729995' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi016-74508-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T04:20:10.917 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:10 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3878746505' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi016-74781-6", "pg_num": 4}]: dispatch 2024-03-26T04:20:10.917 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:10 smithi016 ceph-mon[29101]: 19.1 deep-scrub starts 2024-03-26T04:20:10.918 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:10 smithi016 ceph-mon[29101]: 19.1 deep-scrub ok 2024-03-26T04:20:10.918 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:10 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:10.919 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:10 smithi016 ceph-mon[29101]: osdmap e85: 8 total, 8 up, 8 in 2024-03-26T04:20:10.919 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:10 smithi016 ceph-mon[29101]: from='client.35309 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi016-75529-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi016-75529-2"}]: dispatch 2024-03-26T04:20:10.919 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:10 smithi016 ceph-mon[29101]: from='client.35246 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi016-74508-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T04:20:10.919 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:10 smithi016 ceph-mon[29101]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi016-74781-6", "pg_num": 4}]: dispatch 2024-03-26T04:20:10.919 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:10 smithi016 ceph-mon[29101]: from='client.25282 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi016-74670-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi016-74670-7"}]': finished 2024-03-26T04:20:10.919 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:10 smithi016 ceph-mon[29101]: from='client.25288 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi016-74313-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi016-74313-22"}]': finished 2024-03-26T04:20:10.919 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:10 smithi016 ceph-mon[29101]: from='client.25291 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi016-74362-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi016-74362-10"}]': finished 2024-03-26T04:20:10.919 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:10 smithi016 ceph-mon[29101]: from='client.25294 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi016-74295-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi016-74295-16"}]': finished 2024-03-26T04:20:10.919 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:10 smithi016 ceph-mon[29101]: from='client.25306 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi016-74367-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi016-74367-10"}]': finished 2024-03-26T04:20:10.919 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:10 smithi016 ceph-mon[29101]: osdmap e86: 8 total, 8 up, 8 in 2024-03-26T04:20:10.919 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:10 smithi016 ceph-mon[29101]: Health check update: 9 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:20:11.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:10 smithi183 ceph-mon[33745]: 19.4 deep-scrub starts 2024-03-26T04:20:11.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:10 smithi183 ceph-mon[33745]: 19.4 deep-scrub ok 2024-03-26T04:20:11.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:10 smithi183 ceph-mon[33745]: from='client.35309 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi016-75529-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:20:11.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:10 smithi183 ceph-mon[33745]: from='client.25210 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi016-74285-3", "field": "max_bytes", "val": "4096"}]': finished 2024-03-26T04:20:11.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:10 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/287729995' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi016-74508-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T04:20:11.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:10 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3878746505' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi016-74781-6", "pg_num": 4}]: dispatch 2024-03-26T04:20:11.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:10 smithi183 ceph-mon[33745]: 19.1 deep-scrub starts 2024-03-26T04:20:11.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:10 smithi183 ceph-mon[33745]: 19.1 deep-scrub ok 2024-03-26T04:20:11.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:10 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:11.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:10 smithi183 ceph-mon[33745]: osdmap e85: 8 total, 8 up, 8 in 2024-03-26T04:20:11.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:10 smithi183 ceph-mon[33745]: from='client.35309 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi016-75529-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi016-75529-2"}]: dispatch 2024-03-26T04:20:11.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:10 smithi183 ceph-mon[33745]: from='client.35246 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi016-74508-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T04:20:11.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:10 smithi183 ceph-mon[33745]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi016-74781-6", "pg_num": 4}]: dispatch 2024-03-26T04:20:11.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:10 smithi183 ceph-mon[33745]: from='client.25282 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi016-74670-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi016-74670-7"}]': finished 2024-03-26T04:20:11.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:10 smithi183 ceph-mon[33745]: from='client.25288 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi016-74313-22", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi016-74313-22"}]': finished 2024-03-26T04:20:11.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:10 smithi183 ceph-mon[33745]: from='client.25291 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi016-74362-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi016-74362-10"}]': finished 2024-03-26T04:20:11.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:10 smithi183 ceph-mon[33745]: from='client.25294 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi016-74295-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi016-74295-16"}]': finished 2024-03-26T04:20:11.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:10 smithi183 ceph-mon[33745]: from='client.25306 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi016-74367-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi016-74367-10"}]': finished 2024-03-26T04:20:11.044 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:10 smithi183 ceph-mon[33745]: osdmap e86: 8 total, 8 up, 8 in 2024-03-26T04:20:11.044 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:10 smithi183 ceph-mon[33745]: Health check update: 9 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:20:11.878 INFO:tasks.workunit.client.0.smithi016.stdout: watch_notify: Running main() from gmock_main.cc 2024-03-26T04:20:11.878 INFO:tasks.workunit.client.0.smithi016.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2024-03-26T04:20:11.878 INFO:tasks.workunit.client.0.smithi016.stdout: watch_notify: [----------] Global test environment set-up. 2024-03-26T04:20:11.878 INFO:tasks.workunit.client.0.smithi016.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2024-03-26T04:20:11.878 INFO:tasks.workunit.client.0.smithi016.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2024-03-26T04:20:11.878 INFO:tasks.workunit.client.0.smithi016.stdout: watch_notify: handle_notify cookie 93837944442368 notify_id 326417514497 notifier_gid 34883 2024-03-26T04:20:11.879 INFO:tasks.workunit.client.0.smithi016.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (10792 ms) 2024-03-26T04:20:11.879 INFO:tasks.workunit.client.0.smithi016.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2024-03-26T04:20:11.879 INFO:tasks.workunit.client.0.smithi016.stdout: watch_notify: Trying... 2024-03-26T04:20:11.879 INFO:tasks.workunit.client.0.smithi016.stdout: watch_notify: handle_notify cookie 93837957111984 notify_id 352187318273 notifier_gid 25207 2024-03-26T04:20:11.879 INFO:tasks.workunit.client.0.smithi016.stdout: watch_notify: Waiting for 3.000000000s 2024-03-26T04:20:11.879 INFO:tasks.workunit.client.0.smithi016.stdout: watch_notify: Timed out. 2024-03-26T04:20:11.879 INFO:tasks.workunit.client.0.smithi016.stdout: watch_notify: Flushing... 2024-03-26T04:20:11.879 INFO:tasks.workunit.client.0.smithi016.stdout: watch_notify: Flushed... 2024-03-26T04:20:11.879 INFO:tasks.workunit.client.0.smithi016.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (12108 ms) 2024-03-26T04:20:11.879 INFO:tasks.workunit.client.0.smithi016.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (22900 ms total) 2024-03-26T04:20:11.879 INFO:tasks.workunit.client.0.smithi016.stdout: watch_notify: 2024-03-26T04:20:11.879 INFO:tasks.workunit.client.0.smithi016.stdout: watch_notify: [----------] Global test environment tear-down 2024-03-26T04:20:11.879 INFO:tasks.workunit.client.0.smithi016.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (22900 ms total) 2024-03-26T04:20:11.880 INFO:tasks.workunit.client.0.smithi016.stdout: watch_notify: [ PASSED ] 2 tests. 2024-03-26T04:20:11.880 INFO:tasks.workunit.client.0.smithi016.stdout: cmd: Running main() from gmock_main.cc 2024-03-26T04:20:11.880 INFO:tasks.workunit.client.0.smithi016.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2024-03-26T04:20:11.880 INFO:tasks.workunit.client.0.smithi016.stdout: cmd: [----------] Global test environment set-up. 2024-03-26T04:20:11.880 INFO:tasks.workunit.client.0.smithi016.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2024-03-26T04:20:11.880 INFO:tasks.workunit.client.0.smithi016.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2024-03-26T04:20:11.880 INFO:tasks.workunit.client.0.smithi016.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (9988 ms) 2024-03-26T04:20:11.880 INFO:tasks.workunit.client.0.smithi016.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2024-03-26T04:20:11.880 INFO:tasks.workunit.client.0.smithi016.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (4993 ms) 2024-03-26T04:20:11.880 INFO:tasks.workunit.client.0.smithi016.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2024-03-26T04:20:11.880 INFO:tasks.workunit.client.0.smithi016.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (8015 ms) 2024-03-26T04:20:11.880 INFO:tasks.workunit.client.0.smithi016.stdout: cmd: [----------] 3 tests from NeoRadosCmd (22996 ms total) 2024-03-26T04:20:11.880 INFO:tasks.workunit.client.0.smithi016.stdout: cmd: 2024-03-26T04:20:11.881 INFO:tasks.workunit.client.0.smithi016.stdout: cmd: [----------] Global test environment tear-down 2024-03-26T04:20:11.881 INFO:tasks.workunit.client.0.smithi016.stdout: cmd: [==========] 3 tests from 1 test suite ran. (22996 ms total) 2024-03-26T04:20:11.881 INFO:tasks.workunit.client.0.smithi016.stdout: cmd: [ PASSED ] 3 tests. 2024-03-26T04:20:11.918 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[29101]: pgmap v85: 844 pgs: 174 creating+peering, 630 active+clean, 40 unknown; 145 MiB data, 768 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 3.0 KiB/s wr, 5 op/s 2024-03-26T04:20:11.918 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[29101]: pgmap v87: 788 pgs: 80 creating+peering, 564 active+clean, 144 unknown; 145 MiB data, 768 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T04:20:11.918 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[29101]: from='client.35246 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi016-74508-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-26T04:20:11.918 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[29101]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi016-74781-6", "pg_num": 4}]': finished 2024-03-26T04:20:11.918 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3878746505' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi016-74781-4", "tierpool": "test-rados-api-smithi016-74781-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:20:11.919 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/287729995' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi016-74508-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T04:20:11.919 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3740118015' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi016-75488-2"}]: dispatch 2024-03-26T04:20:11.919 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/4045889979' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi016-74578-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:11.919 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:11.919 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[29101]: osdmap e87: 8 total, 8 up, 8 in 2024-03-26T04:20:11.919 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[29101]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi016-74781-4", "tierpool": "test-rados-api-smithi016-74781-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:20:11.919 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[29101]: from='client.35246 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi016-74508-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T04:20:11.920 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[29101]: from='client.35264 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi016-75488-2"}]: dispatch 2024-03-26T04:20:11.920 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[29101]: from='client.35309 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi016-75529-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi016-75529-2"}]': finished 2024-03-26T04:20:11.920 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[29101]: osdmap e88: 8 total, 8 up, 8 in 2024-03-26T04:20:11.920 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[29101]: from='client.35318 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi016-74578-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:11.920 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/374527304' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:11.920 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3661637495' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:11.920 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[29101]: from='client.35339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:11.920 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[29101]: from='client.35348 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:11.920 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[29101]: from='client.25081 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-4-cache", "mode": "writeback"}]: dispatch 2024-03-26T04:20:11.920 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[29101]: pgmap v91: 928 pgs: 64 creating+peering, 548 active+clean, 316 unknown; 145 MiB data, 768 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:20:11.921 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:11.921 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[37215]: pgmap v85: 844 pgs: 174 creating+peering, 630 active+clean, 40 unknown; 145 MiB data, 768 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 3.0 KiB/s wr, 5 op/s 2024-03-26T04:20:11.921 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[37215]: pgmap v87: 788 pgs: 80 creating+peering, 564 active+clean, 144 unknown; 145 MiB data, 768 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T04:20:11.921 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[37215]: from='client.35246 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi016-74508-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-26T04:20:11.921 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[37215]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi016-74781-6", "pg_num": 4}]': finished 2024-03-26T04:20:11.921 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3878746505' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi016-74781-4", "tierpool": "test-rados-api-smithi016-74781-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:20:11.922 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/287729995' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi016-74508-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T04:20:11.922 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3740118015' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi016-75488-2"}]: dispatch 2024-03-26T04:20:11.922 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/4045889979' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi016-74578-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:11.922 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:11.922 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[37215]: osdmap e87: 8 total, 8 up, 8 in 2024-03-26T04:20:11.922 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[37215]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi016-74781-4", "tierpool": "test-rados-api-smithi016-74781-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:20:11.923 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[37215]: from='client.35246 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi016-74508-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T04:20:11.923 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[37215]: from='client.35264 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi016-75488-2"}]: dispatch 2024-03-26T04:20:11.923 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[37215]: from='client.35309 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi016-75529-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi016-75529-2"}]': finished 2024-03-26T04:20:11.923 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[37215]: osdmap e88: 8 total, 8 up, 8 in 2024-03-26T04:20:11.923 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[37215]: from='client.35318 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi016-74578-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:11.923 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/374527304' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:11.923 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3661637495' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:11.924 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[37215]: from='client.35339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:11.924 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[37215]: from='client.35348 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:11.924 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[37215]: from='client.25081 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-4-cache", "mode": "writeback"}]: dispatch 2024-03-26T04:20:11.924 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[37215]: pgmap v91: 928 pgs: 64 creating+peering, 548 active+clean, 316 unknown; 145 MiB data, 768 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:20:11.924 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:11 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:12.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:11 smithi183 ceph-mon[33745]: pgmap v85: 844 pgs: 174 creating+peering, 630 active+clean, 40 unknown; 145 MiB data, 768 MiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 3.0 KiB/s wr, 5 op/s 2024-03-26T04:20:12.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:11 smithi183 ceph-mon[33745]: pgmap v87: 788 pgs: 80 creating+peering, 564 active+clean, 144 unknown; 145 MiB data, 768 MiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T04:20:12.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:11 smithi183 ceph-mon[33745]: from='client.35246 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi016-74508-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-26T04:20:12.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:11 smithi183 ceph-mon[33745]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi016-74781-6", "pg_num": 4}]': finished 2024-03-26T04:20:12.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:11 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3878746505' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi016-74781-4", "tierpool": "test-rados-api-smithi016-74781-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:20:12.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:11 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/287729995' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi016-74508-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T04:20:12.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:11 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3740118015' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi016-75488-2"}]: dispatch 2024-03-26T04:20:12.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:11 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/4045889979' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi016-74578-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:12.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:11 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:12.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:11 smithi183 ceph-mon[33745]: osdmap e87: 8 total, 8 up, 8 in 2024-03-26T04:20:12.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:11 smithi183 ceph-mon[33745]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi016-74781-4", "tierpool": "test-rados-api-smithi016-74781-6", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:20:12.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:11 smithi183 ceph-mon[33745]: from='client.35246 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi016-74508-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T04:20:12.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:11 smithi183 ceph-mon[33745]: from='client.35264 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi016-75488-2"}]: dispatch 2024-03-26T04:20:12.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:11 smithi183 ceph-mon[33745]: from='client.35309 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsNSsmithi016-75529-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi016-75529-2"}]': finished 2024-03-26T04:20:12.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:11 smithi183 ceph-mon[33745]: osdmap e88: 8 total, 8 up, 8 in 2024-03-26T04:20:12.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:11 smithi183 ceph-mon[33745]: from='client.35318 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi016-74578-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:12.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:11 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/374527304' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:12.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:11 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3661637495' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:12.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:11 smithi183 ceph-mon[33745]: from='client.35339 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:12.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:11 smithi183 ceph-mon[33745]: from='client.35348 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:12.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:11 smithi183 ceph-mon[33745]: from='client.25081 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-4-cache", "mode": "writeback"}]: dispatch 2024-03-26T04:20:12.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:11 smithi183 ceph-mon[33745]: pgmap v91: 928 pgs: 64 creating+peering, 548 active+clean, 316 unknown; 145 MiB data, 768 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:20:12.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:11 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:12.068 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: Running main() from gmock_main.cc 2024-03-26T04:20:12.068 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2024-03-26T04:20:12.068 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [----------] Global test environment set-up. 2024-03-26T04:20:12.068 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2024-03-26T04:20:12.069 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: seed 74313 2024-03-26T04:20:12.069 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-03-26T04:20:12.069 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-03-26T04:20:12.069 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-03-26T04:20:12.069 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (344 ms) 2024-03-26T04:20:12.069 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-03-26T04:20:12.069 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (16 ms) 2024-03-26T04:20:12.070 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-03-26T04:20:12.070 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (6 ms) 2024-03-26T04:20:12.070 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-03-26T04:20:12.070 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (6 ms) 2024-03-26T04:20:12.070 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-03-26T04:20:12.070 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (5 ms) 2024-03-26T04:20:12.071 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-03-26T04:20:12.071 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (3 ms) 2024-03-26T04:20:12.071 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-03-26T04:20:12.071 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (4 ms) 2024-03-26T04:20:12.071 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-03-26T04:20:12.071 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (6 ms) 2024-03-26T04:20:12.071 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-03-26T04:20:12.071 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (6 ms) 2024-03-26T04:20:12.072 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-03-26T04:20:12.072 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (3 ms) 2024-03-26T04:20:12.072 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-03-26T04:20:12.072 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (4 ms) 2024-03-26T04:20:12.072 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-03-26T04:20:12.072 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (5 ms) 2024-03-26T04:20:12.072 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-03-26T04:20:12.072 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (6 ms) 2024-03-26T04:20:12.072 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-03-26T04:20:12.072 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (5 ms) 2024-03-26T04:20:12.073 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-03-26T04:20:12.073 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (16 ms) 2024-03-26T04:20:12.073 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-03-26T04:20:12.073 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (6 ms) 2024-03-26T04:20:12.073 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-03-26T04:20:12.073 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (4 ms) 2024-03-26T04:20:12.073 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-03-26T04:20:12.073 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (2 ms) 2024-03-26T04:20:12.073 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-03-26T04:20:12.073 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (4 ms) 2024-03-26T04:20:12.073 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (451 ms total) 2024-03-26T04:20:12.073 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: 2024-03-26T04:20:12.073 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-03-26T04:20:12.073 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-03-26T04:20:12.073 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (2720 ms) 2024-03-26T04:20:12.074 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-03-26T04:20:12.074 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (13 ms) 2024-03-26T04:20:12.074 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-03-26T04:20:12.074 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (3 ms) 2024-03-26T04:20:12.074 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-03-26T04:20:12.074 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (4 ms) 2024-03-26T04:20:12.074 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-03-26T04:20:12.074 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (2 ms) 2024-03-26T04:20:12.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:20:12 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:20:12.881 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ RUN ] ec_io: Running main() from gmock_main.cc 2024-03-26T04:20:12.881 INFO:tasks.workunit.client.0.smithi016.stdout: ec_io: [==========] Running 2 tests from 1 test suite. 2024-03-26T04:20:12.881 INFO:tasks.workunit.client.0.smithi016.stdout: ec_io: [----------] Global test environment set-up. 2024-03-26T04:20:12.881 INFO:tasks.workunit.client.0.smithi016.stdout: ec_io: [----------] 2 tests from NeoRadosECIo 2024-03-26T04:20:12.881 INFO:tasks.workunit.client.0.smithi016.stdout: ec_io: [ RUN ] NeoRadosECIo.SimpleWrite 2024-03-26T04:20:12.881 INFO:tasks.workunit.client.0.smithi016.stdout: ec_io: [ OK ] NeoRadosECIo.SimpleWrite (14906 ms) 2024-03-26T04:20:12.881 INFO:tasks.workunit.client.0.smithi016.stdout: ec_io: [ RUN ] NeoRadosECIo.ReadOp 2024-03-26T04:20:12.881 INFO:tasks.workunit.client.0.smithi016.stdout: ec_io: [ OK ] NeoRadosECIo.ReadOp (9029 ms) 2024-03-26T04:20:12.881 INFO:tasks.workunit.client.0.smithi016.stdout: ec_io: [----------] 2 tests from NeoRadosECIo (23935 ms total) 2024-03-26T04:20:12.881 INFO:tasks.workunit.client.0.smithi016.stdout: ec_io: 2024-03-26T04:20:12.881 INFO:tasks.workunit.client.0.smithi016.stdout: ec_io: [----------] Global test environment tear-down 2024-03-26T04:20:12.881 INFO:tasks.workunit.client.0.smithi016.stdout: ec_io: [==========] 2 tests from 1 test suite ran. (23943 ms total) 2024-03-26T04:20:12.882 INFO:tasks.workunit.client.0.smithi016.stdout: ec_io: [ PASSED ] 2 tests. 2024-03-26T04:20:12.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[37215]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:20:12.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[37215]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi016-74781-4", "tierpool": "test-rados-api-smithi016-74781-6", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:20:12.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[37215]: from='client.35246 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi016-74508-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-26T04:20:12.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[37215]: from='client.35264 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi016-75488-2"}]': finished 2024-03-26T04:20:12.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[37215]: from='client.35318 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi016-74578-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:12.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[37215]: from='client.35339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:12.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[37215]: from='client.35348 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:12.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[37215]: from='client.25081 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-4-cache", "mode": "writeback"}]': finished 2024-03-26T04:20:12.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[37215]: osdmap e89: 8 total, 8 up, 8 in 2024-03-26T04:20:12.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3740118015' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi016-75488-2"}]: dispatch 2024-03-26T04:20:12.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3878746505' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi016-74781-4", "overlaypool": "test-rados-api-smithi016-74781-6"}]: dispatch 2024-03-26T04:20:12.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3374323318' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi016-74264-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:12.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/287729995' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi016-74508-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T04:20:12.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[37215]: from='client.35264 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi016-75488-2"}]: dispatch 2024-03-26T04:20:12.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[37215]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi016-74781-4", "overlaypool": "test-rados-api-smithi016-74781-6"}]: dispatch 2024-03-26T04:20:12.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[37215]: from='client.35246 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi016-74508-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T04:20:12.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/4135966355' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi016-74841-12"}]: dispatch 2024-03-26T04:20:12.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:20:12.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1973575539' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-4"}]: dispatch 2024-03-26T04:20:12.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[29101]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:20:12.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[29101]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi016-74781-4", "tierpool": "test-rados-api-smithi016-74781-6", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:20:12.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[29101]: from='client.35246 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi016-74508-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-26T04:20:12.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[29101]: from='client.35264 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi016-75488-2"}]': finished 2024-03-26T04:20:12.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[29101]: from='client.35318 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi016-74578-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:12.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[29101]: from='client.35339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:12.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[29101]: from='client.35348 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:12.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[29101]: from='client.25081 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-4-cache", "mode": "writeback"}]': finished 2024-03-26T04:20:12.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[29101]: osdmap e89: 8 total, 8 up, 8 in 2024-03-26T04:20:12.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3740118015' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi016-75488-2"}]: dispatch 2024-03-26T04:20:12.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3878746505' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi016-74781-4", "overlaypool": "test-rados-api-smithi016-74781-6"}]: dispatch 2024-03-26T04:20:12.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3374323318' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi016-74264-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:12.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/287729995' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi016-74508-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T04:20:12.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[29101]: from='client.35264 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi016-75488-2"}]: dispatch 2024-03-26T04:20:12.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[29101]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi016-74781-4", "overlaypool": "test-rados-api-smithi016-74781-6"}]: dispatch 2024-03-26T04:20:12.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[29101]: from='client.35246 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi016-74508-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T04:20:12.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/4135966355' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi016-74841-12"}]: dispatch 2024-03-26T04:20:12.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:20:12.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1973575539' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-4"}]: dispatch 2024-03-26T04:20:12.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:12.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[37215]: from='client.25414 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi016-74264-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:12.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[37215]: from='client.35360 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi016-74841-12"}]: dispatch 2024-03-26T04:20:12.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/4135966355' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi016-74841-12"}]: dispatch 2024-03-26T04:20:12.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[37215]: from='client.35360 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi016-74841-12"}]: dispatch 2024-03-26T04:20:12.917 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/4135966355' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi016-74841-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:12.917 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[37215]: from='client.35360 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi016-74841-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:12.917 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[37215]: from='client.25081 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-4"}]: dispatch 2024-03-26T04:20:12.917 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:12.917 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[29101]: from='client.25414 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi016-74264-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:12.917 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[29101]: from='client.35360 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi016-74841-12"}]: dispatch 2024-03-26T04:20:12.917 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/4135966355' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi016-74841-12"}]: dispatch 2024-03-26T04:20:12.917 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[29101]: from='client.35360 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi016-74841-12"}]: dispatch 2024-03-26T04:20:12.917 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/4135966355' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi016-74841-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:12.917 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[29101]: from='client.35360 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi016-74841-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:12.918 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:12 smithi016 ceph-mon[29101]: from='client.25081 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-4"}]: dispatch 2024-03-26T04:20:13.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:12 smithi183 ceph-mon[33745]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:20:13.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:12 smithi183 ceph-mon[33745]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi016-74781-4", "tierpool": "test-rados-api-smithi016-74781-6", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:20:13.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:12 smithi183 ceph-mon[33745]: from='client.35246 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMisc_smithi016-74508-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-26T04:20:13.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:12 smithi183 ceph-mon[33745]: from='client.35264 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReadOpsmithi016-75488-2"}]': finished 2024-03-26T04:20:13.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:12 smithi183 ceph-mon[33745]: from='client.35318 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManaged_smithi016-74578-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:13.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:12 smithi183 ceph-mon[33745]: from='client.35339 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:13.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:12 smithi183 ceph-mon[33745]: from='client.35348 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:13.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:12 smithi183 ceph-mon[33745]: from='client.25081 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-4-cache", "mode": "writeback"}]': finished 2024-03-26T04:20:13.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:12 smithi183 ceph-mon[33745]: osdmap e89: 8 total, 8 up, 8 in 2024-03-26T04:20:13.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:12 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3740118015' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi016-75488-2"}]: dispatch 2024-03-26T04:20:13.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:12 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3878746505' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi016-74781-4", "overlaypool": "test-rados-api-smithi016-74781-6"}]: dispatch 2024-03-26T04:20:13.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:12 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3374323318' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi016-74264-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:13.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:12 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/287729995' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi016-74508-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T04:20:13.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:12 smithi183 ceph-mon[33745]: from='client.35264 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi016-75488-2"}]: dispatch 2024-03-26T04:20:13.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:12 smithi183 ceph-mon[33745]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi016-74781-4", "overlaypool": "test-rados-api-smithi016-74781-6"}]: dispatch 2024-03-26T04:20:13.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:12 smithi183 ceph-mon[33745]: from='client.35246 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi016-74508-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T04:20:13.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:12 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/4135966355' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi016-74841-12"}]: dispatch 2024-03-26T04:20:13.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:12 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:20:13.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:12 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1973575539' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-4"}]: dispatch 2024-03-26T04:20:13.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:12 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:13.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:12 smithi183 ceph-mon[33745]: from='client.25414 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi016-74264-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:13.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:12 smithi183 ceph-mon[33745]: from='client.35360 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi016-74841-12"}]: dispatch 2024-03-26T04:20:13.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:12 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/4135966355' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi016-74841-12"}]: dispatch 2024-03-26T04:20:13.044 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:12 smithi183 ceph-mon[33745]: from='client.35360 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi016-74841-12"}]: dispatch 2024-03-26T04:20:13.044 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:12 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/4135966355' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi016-74841-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:13.044 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:12 smithi183 ceph-mon[33745]: from='client.35360 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi016-74841-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:13.044 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:12 smithi183 ceph-mon[33745]: from='client.25081 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-4"}]: dispatch 2024-03-26T04:20:14.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:13 smithi183 ceph-mon[33745]: from='client.35264 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi016-75488-2"}]': finished 2024-03-26T04:20:14.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:13 smithi183 ceph-mon[33745]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi016-74781-4", "overlaypool": "test-rados-api-smithi016-74781-6"}]': finished 2024-03-26T04:20:14.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:13 smithi183 ceph-mon[33745]: from='client.35246 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi016-74508-1","app":"app1","key":"key1"}]': finished 2024-03-26T04:20:14.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:13 smithi183 ceph-mon[33745]: from='client.25414 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi016-74264-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:14.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:13 smithi183 ceph-mon[33745]: from='client.35360 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi016-74841-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:20:14.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:13 smithi183 ceph-mon[33745]: from='client.25081 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-4"}]': finished 2024-03-26T04:20:14.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:13 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1973575539' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-4", "tierpool": "test-rados-api-smithi016-74534-4-cache"}]: dispatch 2024-03-26T04:20:14.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:13 smithi183 ceph-mon[33745]: osdmap e90: 8 total, 8 up, 8 in 2024-03-26T04:20:14.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:13 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3878746505' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74781-6", "mode": "writeback"}]: dispatch 2024-03-26T04:20:14.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:13 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/4135966355' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi016-74841-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi016-74841-12"}]: dispatch 2024-03-26T04:20:14.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:13 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/806805394' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi016-74313-22"}]: dispatch 2024-03-26T04:20:14.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:13 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3172019969' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi016-74295-16"}]: dispatch 2024-03-26T04:20:14.295 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:13 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2267327916' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi016-74692-7"}]: dispatch 2024-03-26T04:20:14.295 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:13 smithi183 ceph-mon[33745]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74781-6", "mode": "writeback"}]: dispatch 2024-03-26T04:20:14.295 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:13 smithi183 ceph-mon[33745]: from='client.25081 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-4", "tierpool": "test-rados-api-smithi016-74534-4-cache"}]: dispatch 2024-03-26T04:20:14.295 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:13 smithi183 ceph-mon[33745]: from='client.35360 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi016-74841-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi016-74841-12"}]: dispatch 2024-03-26T04:20:14.295 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:13 smithi183 ceph-mon[33745]: from='client.25288 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi016-74313-22"}]: dispatch 2024-03-26T04:20:14.295 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:13 smithi183 ceph-mon[33745]: from='client.25294 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi016-74295-16"}]: dispatch 2024-03-26T04:20:14.295 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:13 smithi183 ceph-mon[33745]: from='client.25270 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi016-74692-7"}]: dispatch 2024-03-26T04:20:14.296 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:13 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1385062143' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi016-74670-7"}]: dispatch 2024-03-26T04:20:14.296 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:13 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/929019900' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi016-75529-2"}]: dispatch 2024-03-26T04:20:14.296 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:13 smithi183 ceph-mon[33745]: from='client.25282 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi016-74670-7"}]: dispatch 2024-03-26T04:20:14.296 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:13 smithi183 ceph-mon[33745]: from='client.35309 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi016-75529-2"}]: dispatch 2024-03-26T04:20:14.296 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:13 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:14.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[37215]: from='client.35264 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi016-75488-2"}]': finished 2024-03-26T04:20:14.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[37215]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi016-74781-4", "overlaypool": "test-rados-api-smithi016-74781-6"}]': finished 2024-03-26T04:20:14.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[37215]: from='client.35246 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi016-74508-1","app":"app1","key":"key1"}]': finished 2024-03-26T04:20:14.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[37215]: from='client.25414 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi016-74264-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:14.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[37215]: from='client.35360 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi016-74841-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:20:14.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[37215]: from='client.25081 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-4"}]': finished 2024-03-26T04:20:14.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1973575539' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-4", "tierpool": "test-rados-api-smithi016-74534-4-cache"}]: dispatch 2024-03-26T04:20:14.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[37215]: osdmap e90: 8 total, 8 up, 8 in 2024-03-26T04:20:14.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3878746505' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74781-6", "mode": "writeback"}]: dispatch 2024-03-26T04:20:14.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/4135966355' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi016-74841-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi016-74841-12"}]: dispatch 2024-03-26T04:20:14.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/806805394' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi016-74313-22"}]: dispatch 2024-03-26T04:20:14.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3172019969' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi016-74295-16"}]: dispatch 2024-03-26T04:20:14.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2267327916' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi016-74692-7"}]: dispatch 2024-03-26T04:20:14.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[37215]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74781-6", "mode": "writeback"}]: dispatch 2024-03-26T04:20:14.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[37215]: from='client.25081 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-4", "tierpool": "test-rados-api-smithi016-74534-4-cache"}]: dispatch 2024-03-26T04:20:14.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[37215]: from='client.35360 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi016-74841-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi016-74841-12"}]: dispatch 2024-03-26T04:20:14.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[37215]: from='client.25288 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi016-74313-22"}]: dispatch 2024-03-26T04:20:14.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[37215]: from='client.25294 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi016-74295-16"}]: dispatch 2024-03-26T04:20:14.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[37215]: from='client.25270 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi016-74692-7"}]: dispatch 2024-03-26T04:20:14.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1385062143' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi016-74670-7"}]: dispatch 2024-03-26T04:20:14.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/929019900' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi016-75529-2"}]: dispatch 2024-03-26T04:20:14.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[37215]: from='client.25282 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi016-74670-7"}]: dispatch 2024-03-26T04:20:14.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[37215]: from='client.35309 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi016-75529-2"}]: dispatch 2024-03-26T04:20:14.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:14.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[29101]: from='client.35264 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReadOpsmithi016-75488-2"}]': finished 2024-03-26T04:20:14.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[29101]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi016-74781-4", "overlaypool": "test-rados-api-smithi016-74781-6"}]': finished 2024-03-26T04:20:14.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[29101]: from='client.35246 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMisc_smithi016-74508-1","app":"app1","key":"key1"}]': finished 2024-03-26T04:20:14.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[29101]: from='client.25414 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip_smithi016-74264-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:14.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[29101]: from='client.35360 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosWatchNotifyEC_smithi016-74841-12", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:20:14.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[29101]: from='client.25081 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-4"}]': finished 2024-03-26T04:20:14.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1973575539' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-4", "tierpool": "test-rados-api-smithi016-74534-4-cache"}]: dispatch 2024-03-26T04:20:14.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[29101]: osdmap e90: 8 total, 8 up, 8 in 2024-03-26T04:20:14.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3878746505' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74781-6", "mode": "writeback"}]: dispatch 2024-03-26T04:20:14.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/4135966355' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi016-74841-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi016-74841-12"}]: dispatch 2024-03-26T04:20:14.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/806805394' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi016-74313-22"}]: dispatch 2024-03-26T04:20:14.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3172019969' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi016-74295-16"}]: dispatch 2024-03-26T04:20:14.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2267327916' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi016-74692-7"}]: dispatch 2024-03-26T04:20:14.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[29101]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74781-6", "mode": "writeback"}]: dispatch 2024-03-26T04:20:14.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[29101]: from='client.25081 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-4", "tierpool": "test-rados-api-smithi016-74534-4-cache"}]: dispatch 2024-03-26T04:20:14.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[29101]: from='client.35360 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi016-74841-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi016-74841-12"}]: dispatch 2024-03-26T04:20:14.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[29101]: from='client.25288 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi016-74313-22"}]: dispatch 2024-03-26T04:20:14.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[29101]: from='client.25294 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi016-74295-16"}]: dispatch 2024-03-26T04:20:14.418 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[29101]: from='client.25270 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi016-74692-7"}]: dispatch 2024-03-26T04:20:14.418 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1385062143' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi016-74670-7"}]: dispatch 2024-03-26T04:20:14.418 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/929019900' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi016-75529-2"}]: dispatch 2024-03-26T04:20:14.418 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[29101]: from='client.25282 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi016-74670-7"}]: dispatch 2024-03-26T04:20:14.418 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[29101]: from='client.35309 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi016-75529-2"}]: dispatch 2024-03-26T04:20:14.418 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:13 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:15.390 INFO:tasks.workunit.client.0.smithi016.stdout: LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-03-26T04:20:15.390 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (4 ms) 2024-03-26T04:20:15.390 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-03-26T04:20:15.390 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (4 ms) 2024-03-26T04:20:15.390 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-03-26T04:20:15.390 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (3 ms) 2024-03-26T04:20:15.390 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-03-26T04:20:15.390 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (3 ms) 2024-03-26T04:20:15.390 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-03-26T04:20:15.390 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (4 ms) 2024-03-26T04:20:15.390 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-03-26T04:20:15.391 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (2 ms) 2024-03-26T04:20:15.391 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-03-26T04:20:15.391 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (3 ms) 2024-03-26T04:20:15.391 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-03-26T04:20:15.391 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (8 ms) 2024-03-26T04:20:15.391 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-03-26T04:20:15.391 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (3 ms) 2024-03-26T04:20:15.391 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-03-26T04:20:15.391 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (8 ms) 2024-03-26T04:20:15.391 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-03-26T04:20:15.391 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (2 ms) 2024-03-26T04:20:15.391 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-03-26T04:20:15.391 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (4 ms) 2024-03-26T04:20:15.391 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (2790 ms total) 2024-03-26T04:20:15.391 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: 2024-03-26T04:20:15.392 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [----------] Global test environment tear-down 2024-03-26T04:20:15.392 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (26842 ms total) 2024-03-26T04:20:15.392 INFO:tasks.workunit.client.0.smithi016.stdout: api_io_pp: [ PASSED ] 37 tests. 2024-03-26T04:20:15.405 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-03-26T04:20:15.405 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-03-26T04:20:15.405 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-03-26T04:20:15.405 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-03-26T04:20:15.405 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat_pp: seed 74692 2024-03-26T04:20:15.405 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-03-26T04:20:15.405 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (285 ms) 2024-03-26T04:20:15.406 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-03-26T04:20:15.406 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (36 ms) 2024-03-26T04:20:15.406 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-03-26T04:20:15.406 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (0 ms) 2024-03-26T04:20:15.406 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-03-26T04:20:15.406 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (7 ms) 2024-03-26T04:20:15.406 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-03-26T04:20:15.406 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (27 ms) 2024-03-26T04:20:15.406 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (355 ms total) 2024-03-26T04:20:15.406 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat_pp: 2024-03-26T04:20:15.406 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-03-26T04:20:15.406 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-03-26T04:20:15.406 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (3984 ms) 2024-03-26T04:20:15.407 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-03-26T04:20:15.407 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (4 ms) 2024-03-26T04:20:15.407 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-03-26T04:20:15.407 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (12 ms) 2024-03-26T04:20:15.407 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-03-26T04:20:15.407 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (8 ms) 2024-03-26T04:20:15.407 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (4008 ms total) 2024-03-26T04:20:15.407 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat_pp: 2024-03-26T04:20:15.407 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-03-26T04:20:15.407 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (26716 ms total) 2024-03-26T04:20:15.407 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-03-26T04:20:15.408 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: Running main() from gmock_main.cc 2024-03-26T04:20:15.408 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-03-26T04:20:15.408 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [----------] Global test environment set-up. 2024-03-26T04:20:15.408 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-03-26T04:20:15.408 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-03-26T04:20:15.409 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (311 ms) 2024-03-26T04:20:15.409 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-03-26T04:20:15.409 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-03-26T04:20:15.409 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-03-26T04:20:15.409 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: no timeout :/ 2024-03-26T04:20:15.409 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: no timeout :/ 2024-03-26T04:20:15.409 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: no timeout :/ 2024-03-26T04:20:15.409 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: no timeout :/ 2024-03-26T04:20:15.409 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: no timeout :/ 2024-03-26T04:20:15.409 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (63 ms) 2024-03-26T04:20:15.409 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-03-26T04:20:15.409 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (10 ms) 2024-03-26T04:20:15.409 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-03-26T04:20:15.409 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ OK ] LibRadosIo.Checksum (4 ms) 2024-03-26T04:20:15.410 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-03-26T04:20:15.410 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (5 ms) 2024-03-26T04:20:15.410 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-03-26T04:20:15.410 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (6 ms) 2024-03-26T04:20:15.410 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-03-26T04:20:15.410 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (5 ms) 2024-03-26T04:20:15.410 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-03-26T04:20:15.410 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (1 ms) 2024-03-26T04:20:15.410 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-03-26T04:20:15.410 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ OK ] LibRadosIo.TruncTest (4 ms) 2024-03-26T04:20:15.410 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-03-26T04:20:15.410 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (7 ms) 2024-03-26T04:20:15.410 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-03-26T04:20:15.410 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (3 ms) 2024-03-26T04:20:15.411 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-03-26T04:20:15.411 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ OK ] LibRadosIo.RmXattr (16 ms) 2024-03-26T04:20:15.411 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-03-26T04:20:15.411 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ OK ] LibRadosIo.XattrIter (7 ms) 2024-03-26T04:20:15.411 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [----------] 14 tests from LibRadosIo (442 ms total) 2024-03-26T04:20:15.411 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: 2024-03-26T04:20:15.411 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-03-26T04:20:15.411 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-03-26T04:20:15.411 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (2706 ms) 2024-03-26T04:20:15.411 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-03-26T04:20:15.411 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (12 ms) 2024-03-26T04:20:15.411 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-03-26T04:20:15.411 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (7 ms) 2024-03-26T04:20:15.411 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-03-26T04:20:15.411 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (4 ms) 2024-03-26T04:20:15.412 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-03-26T04:20:15.412 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (7 ms) 2024-03-26T04:20:15.412 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-03-26T04:20:15.412 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (4 ms) 2024-03-26T04:20:15.412 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-03-26T04:20:15.412 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (3 ms) 2024-03-26T04:20:15.412 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-03-26T04:20:15.412 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (4 ms) 2024-03-26T04:20:15.412 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-03-26T04:20:15.412 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (10 ms) 2024-03-26T04:20:15.412 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-03-26T04:20:15.413 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (4 ms) 2024-03-26T04:20:15.413 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [----------] 10 tests from LibRadosIoEC (2761 ms total) 2024-03-26T04:20:15.413 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: 2024-03-26T04:20:15.413 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [----------] Global test environment tear-down 2024-03-26T04:20:15.413 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [==========] 24 tests from 2 test suites ran. (26871 ms total) 2024-03-26T04:20:15.413 INFO:tasks.workunit.client.0.smithi016.stdout: api_io: [ PASSED ] 24 tests. 2024-03-26T04:20:15.414 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat: Running main() from gmock_main.cc 2024-03-26T04:20:15.414 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-03-26T04:20:15.414 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat: [----------] Global test environment set-up. 2024-03-26T04:20:15.414 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-03-26T04:20:15.414 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-03-26T04:20:15.414 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat: [ OK ] LibRadosStat.Stat (347 ms) 2024-03-26T04:20:15.414 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-03-26T04:20:15.414 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (6 ms) 2024-03-26T04:20:15.414 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-03-26T04:20:15.414 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat: [ OK ] LibRadosStat.StatNS (19 ms) 2024-03-26T04:20:15.414 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-03-26T04:20:15.414 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (1 ms) 2024-03-26T04:20:15.414 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-03-26T04:20:15.415 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (4 ms) 2024-03-26T04:20:15.415 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat: [----------] 5 tests from LibRadosStat (377 ms total) 2024-03-26T04:20:15.415 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat: 2024-03-26T04:20:15.415 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-03-26T04:20:15.415 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-03-26T04:20:15.415 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (2677 ms) 2024-03-26T04:20:15.415 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-03-26T04:20:15.415 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (25 ms) 2024-03-26T04:20:15.415 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-03-26T04:20:15.415 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (0 ms) 2024-03-26T04:20:15.415 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-03-26T04:20:15.415 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (4 ms) 2024-03-26T04:20:15.415 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (2706 ms total) 2024-03-26T04:20:15.415 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat: 2024-03-26T04:20:15.415 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat: [----------] Global test environment tear-down 2024-03-26T04:20:15.416 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (26724 ms total) 2024-03-26T04:20:15.416 INFO:tasks.workunit.client.0.smithi016.stdout: api_stat: [ PASSED ] 9 tests. 2024-03-26T04:20:16.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:16 smithi016 ceph-mon[29101]: pgmap v94: 728 pgs: 64 creating+peering, 1 active, 20 creating+activating, 611 active+clean, 32 unknown; 145 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T04:20:16.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:16 smithi016 ceph-mon[29101]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74781-6", "mode": "writeback"}]': finished 2024-03-26T04:20:16.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:16 smithi016 ceph-mon[29101]: from='client.25081 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-4", "tierpool": "test-rados-api-smithi016-74534-4-cache"}]': finished 2024-03-26T04:20:16.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:16 smithi016 ceph-mon[29101]: from='client.25288 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi016-74313-22"}]': finished 2024-03-26T04:20:16.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:16 smithi016 ceph-mon[29101]: from='client.25294 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi016-74295-16"}]': finished 2024-03-26T04:20:16.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:16 smithi016 ceph-mon[29101]: from='client.25270 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi016-74692-7"}]': finished 2024-03-26T04:20:16.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:16 smithi016 ceph-mon[29101]: from='client.25282 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi016-74670-7"}]': finished 2024-03-26T04:20:16.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:16 smithi016 ceph-mon[29101]: from='client.35309 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi016-75529-2"}]': finished 2024-03-26T04:20:16.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:16 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/806805394' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi016-74313-22"}]: dispatch 2024-03-26T04:20:16.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:16 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3172019969' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi016-74295-16"}]: dispatch 2024-03-26T04:20:16.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:16 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1385062143' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi016-74670-7"}]: dispatch 2024-03-26T04:20:16.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:16 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/929019900' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi016-75529-2"}]: dispatch 2024-03-26T04:20:16.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:16 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2267327916' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi016-74692-7"}]: dispatch 2024-03-26T04:20:16.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:16 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:16.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:16 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3066044494' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:16.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:16 smithi016 ceph-mon[29101]: osdmap e91: 8 total, 8 up, 8 in 2024-03-26T04:20:16.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:16 smithi016 ceph-mon[29101]: from='client.35309 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi016-75529-2"}]: dispatch 2024-03-26T04:20:16.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:16 smithi016 ceph-mon[29101]: from='client.25288 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi016-74313-22"}]: dispatch 2024-03-26T04:20:16.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:16 smithi016 ceph-mon[37215]: pgmap v94: 728 pgs: 64 creating+peering, 1 active, 20 creating+activating, 611 active+clean, 32 unknown; 145 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T04:20:16.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:16 smithi016 ceph-mon[37215]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74781-6", "mode": "writeback"}]': finished 2024-03-26T04:20:16.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:16 smithi016 ceph-mon[37215]: from='client.25081 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-4", "tierpool": "test-rados-api-smithi016-74534-4-cache"}]': finished 2024-03-26T04:20:16.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:16 smithi016 ceph-mon[37215]: from='client.25288 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi016-74313-22"}]': finished 2024-03-26T04:20:16.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:16 smithi016 ceph-mon[37215]: from='client.25294 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi016-74295-16"}]': finished 2024-03-26T04:20:16.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:16 smithi016 ceph-mon[37215]: from='client.25270 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi016-74692-7"}]': finished 2024-03-26T04:20:16.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:16 smithi016 ceph-mon[37215]: from='client.25282 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi016-74670-7"}]': finished 2024-03-26T04:20:16.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:16 smithi016 ceph-mon[37215]: from='client.35309 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi016-75529-2"}]': finished 2024-03-26T04:20:16.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:16 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/806805394' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi016-74313-22"}]: dispatch 2024-03-26T04:20:16.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:16 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3172019969' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi016-74295-16"}]: dispatch 2024-03-26T04:20:16.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:16 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1385062143' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi016-74670-7"}]: dispatch 2024-03-26T04:20:16.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:16 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/929019900' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi016-75529-2"}]: dispatch 2024-03-26T04:20:16.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:16 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2267327916' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi016-74692-7"}]: dispatch 2024-03-26T04:20:16.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:16 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:16.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:16 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3066044494' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:16.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:16 smithi016 ceph-mon[37215]: osdmap e91: 8 total, 8 up, 8 in 2024-03-26T04:20:16.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:16 smithi016 ceph-mon[37215]: from='client.35309 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi016-75529-2"}]: dispatch 2024-03-26T04:20:16.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:16 smithi016 ceph-mon[37215]: from='client.25288 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi016-74313-22"}]: dispatch 2024-03-26T04:20:16.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:16 smithi183 ceph-mon[33745]: pgmap v94: 728 pgs: 64 creating+peering, 1 active, 20 creating+activating, 611 active+clean, 32 unknown; 145 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T04:20:16.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:16 smithi183 ceph-mon[33745]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74781-6", "mode": "writeback"}]': finished 2024-03-26T04:20:16.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:16 smithi183 ceph-mon[33745]: from='client.25081 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-4", "tierpool": "test-rados-api-smithi016-74534-4-cache"}]': finished 2024-03-26T04:20:16.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:16 smithi183 ceph-mon[33745]: from='client.25288 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi016-74313-22"}]': finished 2024-03-26T04:20:16.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:16 smithi183 ceph-mon[33745]: from='client.25294 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi016-74295-16"}]': finished 2024-03-26T04:20:16.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:16 smithi183 ceph-mon[33745]: from='client.25270 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi016-74692-7"}]': finished 2024-03-26T04:20:16.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:16 smithi183 ceph-mon[33745]: from='client.25282 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi016-74670-7"}]': finished 2024-03-26T04:20:16.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:16 smithi183 ceph-mon[33745]: from='client.35309 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsNSsmithi016-75529-2"}]': finished 2024-03-26T04:20:16.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:16 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/806805394' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi016-74313-22"}]: dispatch 2024-03-26T04:20:16.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:16 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3172019969' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi016-74295-16"}]: dispatch 2024-03-26T04:20:16.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:16 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1385062143' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi016-74670-7"}]: dispatch 2024-03-26T04:20:16.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:16 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/929019900' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi016-75529-2"}]: dispatch 2024-03-26T04:20:16.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:16 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2267327916' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi016-74692-7"}]: dispatch 2024-03-26T04:20:16.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:16 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:16.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:16 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3066044494' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:16.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:16 smithi183 ceph-mon[33745]: osdmap e91: 8 total, 8 up, 8 in 2024-03-26T04:20:16.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:16 smithi183 ceph-mon[33745]: from='client.35309 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi016-75529-2"}]: dispatch 2024-03-26T04:20:16.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:16 smithi183 ceph-mon[33745]: from='client.25288 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi016-74313-22"}]: dispatch 2024-03-26T04:20:17.169 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-03-26T04:20:17.170 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-03-26T04:20:17.170 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-03-26T04:20:17.170 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-03-26T04:20:17.170 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-03-26T04:20:17.170 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: notify 2024-03-26T04:20:17.170 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (2290 ms) 2024-03-26T04:20:17.170 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-03-26T04:20:17.170 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (4 ms) 2024-03-26T04:20:17.170 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (2294 ms total) 2024-03-26T04:20:17.170 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: 2024-03-26T04:20:17.170 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-03-26T04:20:17.170 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-03-26T04:20:17.170 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: notify 2024-03-26T04:20:17.171 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (122 ms) 2024-03-26T04:20:17.171 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-03-26T04:20:17.171 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: notify 2024-03-26T04:20:17.171 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (5893 ms) 2024-03-26T04:20:17.171 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-03-26T04:20:17.171 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (53 ms) 2024-03-26T04:20:17.171 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-03-26T04:20:17.171 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (10 ms) 2024-03-26T04:20:17.171 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-03-26T04:20:17.171 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: handle_notify cookie 94258755783008 notify_id 390842023939 notifier_gid 35288 2024-03-26T04:20:17.171 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (12 ms) 2024-03-26T04:20:17.171 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-03-26T04:20:17.171 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: handle_notify cookie 94258755786128 notify_id 390842023940 notifier_gid 35288 2024-03-26T04:20:17.171 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (10 ms) 2024-03-26T04:20:17.171 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-03-26T04:20:17.171 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: handle_notify cookie 94258755786128 notify_id 390842023937 notifier_gid 35288 2024-03-26T04:20:17.172 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (8 ms) 2024-03-26T04:20:17.172 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-03-26T04:20:17.172 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: handle_notify cookie 94258755786048 notify_id 390842023941 notifier_gid 35288 2024-03-26T04:20:17.172 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (7 ms) 2024-03-26T04:20:17.172 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-03-26T04:20:17.172 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: handle_notify cookie 94258755786048 notify_id 390842023938 notifier_gid 35288 2024-03-26T04:20:17.172 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (10 ms) 2024-03-26T04:20:17.172 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-03-26T04:20:17.172 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: handle_notify cookie 94258755786048 notify_id 390842023939 notifier_gid 35288 2024-03-26T04:20:17.172 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (18 ms) 2024-03-26T04:20:17.172 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-03-26T04:20:17.172 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: trying... 2024-03-26T04:20:17.172 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: handle_notify cookie 94258755786048 notify_id 390842023942 notifier_gid 35288 2024-03-26T04:20:17.172 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: timed out 2024-03-26T04:20:17.173 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: flushing 2024-03-26T04:20:17.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[29101]: pgmap v96: 728 pgs: 64 creating+peering, 19 creating+activating, 549 active+clean, 96 unknown; 145 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 994 B/s rd, 1 op/s 2024-03-26T04:20:17.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[29101]: from='client.25294 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi016-74295-16"}]: dispatch 2024-03-26T04:20:17.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[29101]: from='client.25282 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi016-74670-7"}]: dispatch 2024-03-26T04:20:17.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[29101]: from='client.25270 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi016-74692-7"}]: dispatch 2024-03-26T04:20:17.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[29101]: from='client.25438 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:17.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:17.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[29101]: Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:20:17.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[29101]: pool 'PoolQuotaPP_smithi016-74285-3' is full (reached quota's max_bytes: 4 KiB) 2024-03-26T04:20:17.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[29101]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T04:20:17.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[29101]: from='client.35360 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi016-74841-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi016-74841-12"}]': finished 2024-03-26T04:20:17.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[29101]: from='client.35309 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi016-75529-2"}]': finished 2024-03-26T04:20:17.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[29101]: from='client.25288 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi016-74313-22"}]': finished 2024-03-26T04:20:17.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[29101]: from='client.25294 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi016-74295-16"}]': finished 2024-03-26T04:20:17.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[29101]: from='client.25282 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi016-74670-7"}]': finished 2024-03-26T04:20:17.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[29101]: from='client.25270 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi016-74692-7"}]': finished 2024-03-26T04:20:17.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[29101]: from='client.25438 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:17.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3965498627' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:17.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3019850196' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi016-74362-10"}]: dispatch 2024-03-26T04:20:17.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1871982092' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi016-74367-10"}]: dispatch 2024-03-26T04:20:17.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/4261059399' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi016-75529-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:17.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[29101]: osdmap e92: 8 total, 8 up, 8 in 2024-03-26T04:20:17.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[37215]: pgmap v96: 728 pgs: 64 creating+peering, 19 creating+activating, 549 active+clean, 96 unknown; 145 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 994 B/s rd, 1 op/s 2024-03-26T04:20:17.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[37215]: from='client.25294 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi016-74295-16"}]: dispatch 2024-03-26T04:20:17.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[37215]: from='client.25282 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi016-74670-7"}]: dispatch 2024-03-26T04:20:17.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[37215]: from='client.25270 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi016-74692-7"}]: dispatch 2024-03-26T04:20:17.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[37215]: from='client.25438 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:17.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:17.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[37215]: Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:20:17.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[37215]: pool 'PoolQuotaPP_smithi016-74285-3' is full (reached quota's max_bytes: 4 KiB) 2024-03-26T04:20:17.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[37215]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T04:20:17.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[37215]: from='client.35360 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi016-74841-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi016-74841-12"}]': finished 2024-03-26T04:20:17.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[37215]: from='client.35309 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi016-75529-2"}]': finished 2024-03-26T04:20:17.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[37215]: from='client.25288 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi016-74313-22"}]': finished 2024-03-26T04:20:17.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[37215]: from='client.25294 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi016-74295-16"}]': finished 2024-03-26T04:20:17.666 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[37215]: from='client.25282 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi016-74670-7"}]': finished 2024-03-26T04:20:17.666 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[37215]: from='client.25270 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi016-74692-7"}]': finished 2024-03-26T04:20:17.666 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[37215]: from='client.25438 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:17.666 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3965498627' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:17.666 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3019850196' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi016-74362-10"}]: dispatch 2024-03-26T04:20:17.666 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1871982092' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi016-74367-10"}]: dispatch 2024-03-26T04:20:17.666 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/4261059399' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi016-75529-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:17.666 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[37215]: osdmap e92: 8 total, 8 up, 8 in 2024-03-26T04:20:17.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[29101]: from='client.37634 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi016-75529-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:17.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:17.667 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[29101]: from='client.25447 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:17.667 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[29101]: from='client.37634 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi016-75529-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:20:17.667 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[29101]: from='client.25447 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:17.667 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/4261059399' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi016-75529-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi016-75529-3"}]: dispatch 2024-03-26T04:20:17.667 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[29101]: osdmap e93: 8 total, 8 up, 8 in 2024-03-26T04:20:17.667 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2221359456' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi016-74264-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:17.667 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[29101]: from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi016-74362-10"}]: dispatch 2024-03-26T04:20:17.667 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[29101]: from='client.37634 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi016-75529-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi016-75529-3"}]: dispatch 2024-03-26T04:20:17.667 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[29101]: from='client.37697 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi016-74264-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:17.667 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[29101]: from='client.25306 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi016-74367-10"}]: dispatch 2024-03-26T04:20:17.667 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/900121678' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi016-74609-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:17.667 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[29101]: from='client.26788 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi016-74609-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:17.667 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:17.668 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[37215]: from='client.37634 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi016-75529-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:17.668 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:17.668 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[37215]: from='client.25447 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:17.668 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[37215]: from='client.37634 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi016-75529-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:20:17.668 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[37215]: from='client.25447 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:17.668 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/4261059399' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi016-75529-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi016-75529-3"}]: dispatch 2024-03-26T04:20:17.668 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[37215]: osdmap e93: 8 total, 8 up, 8 in 2024-03-26T04:20:17.668 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2221359456' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi016-74264-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:17.668 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[37215]: from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi016-74362-10"}]: dispatch 2024-03-26T04:20:17.668 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[37215]: from='client.37634 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi016-75529-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi016-75529-3"}]: dispatch 2024-03-26T04:20:17.668 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[37215]: from='client.37697 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi016-74264-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:17.668 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[37215]: from='client.25306 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi016-74367-10"}]: dispatch 2024-03-26T04:20:17.668 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/900121678' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi016-74609-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:17.669 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[37215]: from='client.26788 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi016-74609-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:17.669 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:17 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:17.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:17 smithi183 ceph-mon[33745]: pgmap v96: 728 pgs: 64 creating+peering, 19 creating+activating, 549 active+clean, 96 unknown; 145 MiB data, 756 MiB used, 714 GiB / 715 GiB avail; 994 B/s rd, 1 op/s 2024-03-26T04:20:17.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:17 smithi183 ceph-mon[33745]: from='client.25294 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi016-74295-16"}]: dispatch 2024-03-26T04:20:17.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:17 smithi183 ceph-mon[33745]: from='client.25282 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi016-74670-7"}]: dispatch 2024-03-26T04:20:17.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:17 smithi183 ceph-mon[33745]: from='client.25270 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi016-74692-7"}]: dispatch 2024-03-26T04:20:17.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:17 smithi183 ceph-mon[33745]: from='client.25438 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:17.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:17 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:17.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:17 smithi183 ceph-mon[33745]: Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:20:17.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:17 smithi183 ceph-mon[33745]: pool 'PoolQuotaPP_smithi016-74285-3' is full (reached quota's max_bytes: 4 KiB) 2024-03-26T04:20:17.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:17 smithi183 ceph-mon[33745]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T04:20:17.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:17 smithi183 ceph-mon[33745]: from='client.35360 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyEC_smithi016-74841-12", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyEC_smithi016-74841-12"}]': finished 2024-03-26T04:20:17.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:17 smithi183 ceph-mon[33745]: from='client.35309 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsNSsmithi016-75529-2"}]': finished 2024-03-26T04:20:17.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:17 smithi183 ceph-mon[33745]: from='client.25288 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi016-74313-22"}]': finished 2024-03-26T04:20:17.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:17 smithi183 ceph-mon[33745]: from='client.25294 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi016-74295-16"}]': finished 2024-03-26T04:20:17.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:17 smithi183 ceph-mon[33745]: from='client.25282 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi016-74670-7"}]': finished 2024-03-26T04:20:17.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:17 smithi183 ceph-mon[33745]: from='client.25270 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi016-74692-7"}]': finished 2024-03-26T04:20:17.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:17 smithi183 ceph-mon[33745]: from='client.25438 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:17.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:17 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3965498627' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:17.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:17 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3019850196' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi016-74362-10"}]: dispatch 2024-03-26T04:20:17.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:17 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1871982092' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi016-74367-10"}]: dispatch 2024-03-26T04:20:17.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:17 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/4261059399' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi016-75529-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:17.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:17 smithi183 ceph-mon[33745]: osdmap e92: 8 total, 8 up, 8 in 2024-03-26T04:20:17.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:17 smithi183 ceph-mon[33745]: from='client.37634 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi016-75529-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:17.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:17 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:17.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:17 smithi183 ceph-mon[33745]: from='client.25447 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:17.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:17 smithi183 ceph-mon[33745]: from='client.37634 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsManysmithi016-75529-3", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:20:17.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:17 smithi183 ceph-mon[33745]: from='client.25447 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:17.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:17 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/4261059399' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi016-75529-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi016-75529-3"}]: dispatch 2024-03-26T04:20:17.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:17 smithi183 ceph-mon[33745]: osdmap e93: 8 total, 8 up, 8 in 2024-03-26T04:20:17.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:17 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2221359456' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi016-74264-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:17.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:17 smithi183 ceph-mon[33745]: from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi016-74362-10"}]: dispatch 2024-03-26T04:20:17.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:17 smithi183 ceph-mon[33745]: from='client.37634 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi016-75529-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi016-75529-3"}]: dispatch 2024-03-26T04:20:17.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:17 smithi183 ceph-mon[33745]: from='client.37697 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi016-74264-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:17.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:17 smithi183 ceph-mon[33745]: from='client.25306 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi016-74367-10"}]: dispatch 2024-03-26T04:20:17.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:17 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/900121678' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi016-74609-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:17.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:17 smithi183 ceph-mon[33745]: from='client.26788 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi016-74609-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:17.795 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:17 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:18.649 INFO:tasks.workunit.client.0.smithi016.stdout: ap pool: Running main() from gmock_main.cc 2024-03-26T04:20:18.649 INFO:tasks.workunit.client.0.smithi016.stdout: pool: [==========] Running 6 tests from 1 test suite. 2024-03-26T04:20:18.649 INFO:tasks.workunit.client.0.smithi016.stdout: pool: [----------] Global test environment set-up. 2024-03-26T04:20:18.650 INFO:tasks.workunit.client.0.smithi016.stdout: pool: [----------] 6 tests from NeoRadosPools 2024-03-26T04:20:18.650 INFO:tasks.workunit.client.0.smithi016.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2024-03-26T04:20:18.650 INFO:tasks.workunit.client.0.smithi016.stdout: pool: [ OK ] NeoRadosPools.PoolList (10803 ms) 2024-03-26T04:20:18.650 INFO:tasks.workunit.client.0.smithi016.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2024-03-26T04:20:18.650 INFO:tasks.workunit.client.0.smithi016.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (3046 ms) 2024-03-26T04:20:18.650 INFO:tasks.workunit.client.0.smithi016.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2024-03-26T04:20:18.651 INFO:tasks.workunit.client.0.smithi016.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (4166 ms) 2024-03-26T04:20:18.651 INFO:tasks.workunit.client.0.smithi016.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2024-03-26T04:20:18.651 INFO:tasks.workunit.client.0.smithi016.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (7376 ms) 2024-03-26T04:20:18.651 INFO:tasks.workunit.client.0.smithi016.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2024-03-26T04:20:18.651 INFO:tasks.workunit.client.0.smithi016.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (2184 ms) 2024-03-26T04:20:18.651 INFO:tasks.workunit.client.0.smithi016.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2024-03-26T04:20:18.651 INFO:tasks.workunit.client.0.smithi016.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (2107 ms) 2024-03-26T04:20:18.652 INFO:tasks.workunit.client.0.smithi016.stdout: pool: [----------] 6 tests from NeoRadosPools (29682 ms total) 2024-03-26T04:20:18.652 INFO:tasks.workunit.client.0.smithi016.stdout: pool: 2024-03-26T04:20:18.652 INFO:tasks.workunit.client.0.smithi016.stdout: pool: [----------] Global test environment tear-down 2024-03-26T04:20:18.652 INFO:tasks.workunit.client.0.smithi016.stdout: pool: [==========] 6 tests from 1 test suite ran. (29682 ms total) 2024-03-26T04:20:18.652 INFO:tasks.workunit.client.0.smithi016.stdout: pool: [ PASSED ] 6 tests. 2024-03-26T04:20:18.688 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-03-26T04:20:18.688 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-03-26T04:20:18.688 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-03-26T04:20:18.689 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-03-26T04:20:18.689 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock_pp: seed 74362 2024-03-26T04:20:18.689 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-03-26T04:20:18.689 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (322 ms) 2024-03-26T04:20:18.689 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-03-26T04:20:18.689 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (8 ms) 2024-03-26T04:20:18.689 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-03-26T04:20:18.689 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1009 ms) 2024-03-26T04:20:18.689 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-03-26T04:20:18.689 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1021 ms) 2024-03-26T04:20:18.689 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-03-26T04:20:18.689 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (3 ms) 2024-03-26T04:20:18.689 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-03-26T04:20:18.689 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (4 ms) 2024-03-26T04:20:18.690 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-03-26T04:20:18.690 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (4 ms) 2024-03-26T04:20:18.690 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-03-26T04:20:18.690 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (4 ms) 2024-03-26T04:20:18.690 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (2375 ms total) 2024-03-26T04:20:18.690 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock_pp: 2024-03-26T04:20:18.690 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-03-26T04:20:18.690 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-03-26T04:20:18.690 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (2701 ms) 2024-03-26T04:20:18.690 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-03-26T04:20:18.690 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (17 ms) 2024-03-26T04:20:18.690 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-03-26T04:20:18.690 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1084 ms) 2024-03-26T04:20:18.691 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-03-26T04:20:18.691 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1022 ms) 2024-03-26T04:20:18.691 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-03-26T04:20:18.691 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (4 ms) 2024-03-26T04:20:18.691 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-03-26T04:20:18.691 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (12 ms) 2024-03-26T04:20:18.691 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-03-26T04:20:18.691 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (5 ms) 2024-03-26T04:20:18.691 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-03-26T04:20:18.691 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (4 ms) 2024-03-26T04:20:18.691 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (4849 ms total) 2024-03-26T04:20:18.691 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock_pp: 2024-03-26T04:20:18.691 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-03-26T04:20:18.691 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (30127 ms total) 2024-03-26T04:20:18.691 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-03-26T04:20:18.692 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock: Running main() from gmock_main.cc 2024-03-26T04:20:18.692 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-03-26T04:20:18.692 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock: [----------] Global test environment set-up. 2024-03-26T04:20:18.692 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-03-26T04:20:18.692 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-03-26T04:20:18.692 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (347 ms) 2024-03-26T04:20:18.692 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-03-26T04:20:18.692 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock: [ OK ] LibRadosLock.LockShared (12 ms) 2024-03-26T04:20:18.692 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-03-26T04:20:18.692 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1010 ms) 2024-03-26T04:20:18.693 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-03-26T04:20:18.693 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1007 ms) 2024-03-26T04:20:18.693 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-03-26T04:20:18.693 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (53 ms) 2024-03-26T04:20:18.693 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-03-26T04:20:18.693 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock: [ OK ] LibRadosLock.Unlock (15 ms) 2024-03-26T04:20:18.693 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-03-26T04:20:18.693 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (4 ms) 2024-03-26T04:20:18.693 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-03-26T04:20:18.693 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (48 ms) 2024-03-26T04:20:18.693 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock: [----------] 8 tests from LibRadosLock (2497 ms total) 2024-03-26T04:20:18.693 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock: 2024-03-26T04:20:18.693 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-03-26T04:20:18.693 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-03-26T04:20:18.694 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (2713 ms) 2024-03-26T04:20:18.694 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-03-26T04:20:18.694 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (6 ms) 2024-03-26T04:20:18.694 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-03-26T04:20:18.694 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1075 ms) 2024-03-26T04:20:18.694 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-03-26T04:20:18.694 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1054 ms) 2024-03-26T04:20:18.694 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-03-26T04:20:18.695 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (12 ms) 2024-03-26T04:20:18.695 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-03-26T04:20:18.695 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (10 ms) 2024-03-26T04:20:18.695 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-03-26T04:20:18.695 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (23 ms) 2024-03-26T04:20:18.695 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-03-26T04:20:18.695 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (8 ms) 2024-03-26T04:20:18.695 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (4901 ms total) 2024-03-26T04:20:18.695 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock: 2024-03-26T04:20:18.696 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock: [----------] Global test environment tear-down 2024-03-26T04:20:18.696 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (30116 ms total) 2024-03-26T04:20:18.696 INFO:tasks.workunit.client.0.smithi016.stdout: api_lock: [ PASSED ] 16 tests. 2024-03-26T04:20:18.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:20:18 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:20:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:20:18.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:18 smithi016 ceph-mon[29101]: pgmap v99: 752 pgs: 40 creating+peering, 5 creating+activating, 541 active+clean, 166 unknown; 145 MiB data, 761 MiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 1 op/s 2024-03-26T04:20:18.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:18 smithi016 ceph-mon[29101]: from='client.25291 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi016-74362-10"}]': finished 2024-03-26T04:20:18.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:18 smithi016 ceph-mon[29101]: from='client.37697 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi016-74264-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:18.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:18 smithi016 ceph-mon[29101]: from='client.25306 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi016-74367-10"}]': finished 2024-03-26T04:20:18.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:18 smithi016 ceph-mon[29101]: from='client.26788 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi016-74609-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:18.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:18 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3019850196' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi016-74362-10"}]: dispatch 2024-03-26T04:20:18.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:18 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1871982092' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi016-74367-10"}]: dispatch 2024-03-26T04:20:18.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:18 smithi016 ceph-mon[29101]: osdmap e94: 8 total, 8 up, 8 in 2024-03-26T04:20:18.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:18 smithi016 ceph-mon[29101]: from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi016-74362-10"}]: dispatch 2024-03-26T04:20:18.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:18 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/4135966355' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi016-74841-12"}]: dispatch 2024-03-26T04:20:18.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:18 smithi016 ceph-mon[29101]: from='client.35360 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi016-74841-12"}]: dispatch 2024-03-26T04:20:18.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:18 smithi016 ceph-mon[29101]: from='client.25306 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi016-74367-10"}]: dispatch 2024-03-26T04:20:18.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:18 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:18.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:18 smithi016 ceph-mon[37215]: pgmap v99: 752 pgs: 40 creating+peering, 5 creating+activating, 541 active+clean, 166 unknown; 145 MiB data, 761 MiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 1 op/s 2024-03-26T04:20:18.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:18 smithi016 ceph-mon[37215]: from='client.25291 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi016-74362-10"}]': finished 2024-03-26T04:20:18.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:18 smithi016 ceph-mon[37215]: from='client.37697 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi016-74264-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:18.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:18 smithi016 ceph-mon[37215]: from='client.25306 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi016-74367-10"}]': finished 2024-03-26T04:20:18.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:18 smithi016 ceph-mon[37215]: from='client.26788 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi016-74609-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:18.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:18 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3019850196' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi016-74362-10"}]: dispatch 2024-03-26T04:20:18.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:18 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1871982092' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi016-74367-10"}]: dispatch 2024-03-26T04:20:18.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:18 smithi016 ceph-mon[37215]: osdmap e94: 8 total, 8 up, 8 in 2024-03-26T04:20:18.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:18 smithi016 ceph-mon[37215]: from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi016-74362-10"}]: dispatch 2024-03-26T04:20:18.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:18 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/4135966355' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi016-74841-12"}]: dispatch 2024-03-26T04:20:18.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:18 smithi016 ceph-mon[37215]: from='client.35360 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi016-74841-12"}]: dispatch 2024-03-26T04:20:18.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:18 smithi016 ceph-mon[37215]: from='client.25306 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi016-74367-10"}]: dispatch 2024-03-26T04:20:18.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:18 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:19.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:18 smithi183 ceph-mon[33745]: pgmap v99: 752 pgs: 40 creating+peering, 5 creating+activating, 541 active+clean, 166 unknown; 145 MiB data, 761 MiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 1 op/s 2024-03-26T04:20:19.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:18 smithi183 ceph-mon[33745]: from='client.25291 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi016-74362-10"}]': finished 2024-03-26T04:20:19.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:18 smithi183 ceph-mon[33745]: from='client.37697 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip2_smithi016-74264-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:19.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:18 smithi183 ceph-mon[33745]: from='client.25306 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi016-74367-10"}]': finished 2024-03-26T04:20:19.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:18 smithi183 ceph-mon[33745]: from='client.26788 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsSelfManagedPP_smithi016-74609-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:19.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:18 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3019850196' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi016-74362-10"}]: dispatch 2024-03-26T04:20:19.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:18 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1871982092' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi016-74367-10"}]: dispatch 2024-03-26T04:20:19.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:18 smithi183 ceph-mon[33745]: osdmap e94: 8 total, 8 up, 8 in 2024-03-26T04:20:19.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:18 smithi183 ceph-mon[33745]: from='client.25291 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi016-74362-10"}]: dispatch 2024-03-26T04:20:19.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:18 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/4135966355' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi016-74841-12"}]: dispatch 2024-03-26T04:20:19.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:18 smithi183 ceph-mon[33745]: from='client.35360 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi016-74841-12"}]: dispatch 2024-03-26T04:20:19.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:18 smithi183 ceph-mon[33745]: from='client.25306 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi016-74367-10"}]: dispatch 2024-03-26T04:20:19.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:18 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:19.044 INFO:tasks.workunit.client.0.smithi016.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-03-26T04:20:19.044 INFO:tasks.workunit.client.0.smithi016.stdout: api_service_pp: [----------] Global test environment set-up. 2024-03-26T04:20:19.044 INFO:tasks.workunit.client.0.smithi016.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-03-26T04:20:19.044 INFO:tasks.workunit.client.0.smithi016.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-03-26T04:20:19.044 INFO:tasks.workunit.client.0.smithi016.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5045 ms) 2024-03-26T04:20:19.045 INFO:tasks.workunit.client.0.smithi016.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-03-26T04:20:19.045 INFO:tasks.workunit.client.0.smithi016.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (57 ms) 2024-03-26T04:20:19.045 INFO:tasks.workunit.client.0.smithi016.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-03-26T04:20:19.045 INFO:tasks.workunit.client.0.smithi016.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20029 ms) 2024-03-26T04:20:19.045 INFO:tasks.workunit.client.0.smithi016.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-03-26T04:20:19.045 INFO:tasks.workunit.client.0.smithi016.stdout: api_service_pp: attempt 0 of 20 2024-03-26T04:20:19.045 INFO:tasks.workunit.client.0.smithi016.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (5130 ms) 2024-03-26T04:20:19.045 INFO:tasks.workunit.client.0.smithi016.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (30261 ms total) 2024-03-26T04:20:19.045 INFO:tasks.workunit.client.0.smithi016.stdout: api_service_pp: 2024-03-26T04:20:19.045 INFO:tasks.workunit.client.0.smithi016.stdout: api_service_pp: [----------] Global test environment tear-down 2024-03-26T04:20:19.045 INFO:tasks.workunit.client.0.smithi016.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (30261 ms total) 2024-03-26T04:20:19.045 INFO:tasks.workunit.client.0.smithi016.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-03-26T04:20:19.837 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: Running main() from gmock_main.cc 2024-03-26T04:20:19.837 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-03-26T04:20:19.837 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-03-26T04:20:19.837 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-03-26T04:20:19.837 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-03-26T04:20:19.837 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: watch_notify_test_cb 2024-03-26T04:20:19.837 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (359 ms) 2024-03-26T04:20:19.837 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-03-26T04:20:19.838 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-03-26T04:20:19.838 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94623464271360 err -107 2024-03-26T04:20:19.838 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (1016 ms) 2024-03-26T04:20:19.838 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-03-26T04:20:19.838 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-03-26T04:20:19.838 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94623464273408 err -107 2024-03-26T04:20:19.838 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (1037 ms) 2024-03-26T04:20:19.838 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-03-26T04:20:19.838 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: watch_notify2_test_cb from 34751 notify_id 335007449088 cookie 94623464275824 2024-03-26T04:20:19.838 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (35 ms) 2024-03-26T04:20:19.838 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-03-26T04:20:19.838 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: watch_notify2_test_cb from 34751 notify_id 335007449088 cookie 94623464275824 2024-03-26T04:20:19.838 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (6 ms) 2024-03-26T04:20:19.839 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-03-26T04:20:19.839 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: watch_notify2_test_cb from 34751 notify_id 335007449088 cookie 94623464275760 2024-03-26T04:20:19.839 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (7 ms) 2024-03-26T04:20:19.839 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-03-26T04:20:19.839 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: watch_notify2_test_cb from 34751 notify_id 335007449088 cookie 94623464275696 2024-03-26T04:20:19.839 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: watch_notify2_test_cb from 34751 notify_id 335007449088 cookie 94623464380208 2024-03-26T04:20:19.839 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (63 ms) 2024-03-26T04:20:19.839 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-03-26T04:20:19.839 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: watch_notify2_test_cb from 34751 notify_id 335007449089 cookie 94623464382688 2024-03-26T04:20:19.839 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: watch_notify2_test_cb from 34751 notify_id 339302416386 cookie 94623464382688 2024-03-26T04:20:19.839 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3005 ms) 2024-03-26T04:20:19.839 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-03-26T04:20:19.839 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-03-26T04:20:19.839 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94623464382688 err -107 2024-03-26T04:20:19.840 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: watch_notify2_test_cb from 34751 notify_id 365072220162 cookie 94623464382688 2024-03-26T04:20:19.840 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5007 ms) 2024-03-26T04:20:19.840 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-03-26T04:20:19.840 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-03-26T04:20:19.840 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94623464382688 err -107 2024-03-26T04:20:19.840 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1005 ms) 2024-03-26T04:20:19.840 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (11540 ms total) 2024-03-26T04:20:19.840 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: 2024-03-26T04:20:19.840 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-03-26T04:20:19.840 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-03-26T04:20:19.840 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: watch_notify_test_cb 2024-03-26T04:20:19.840 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (1212 ms) 2024-03-26T04:20:19.840 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (1212 ms total) 2024-03-26T04:20:19.840 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: 2024-03-26T04:20:19.841 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-03-26T04:20:19.841 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (31115 ms total) 2024-03-26T04:20:19.841 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-03-26T04:20:20.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:19 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:20:20.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:19 smithi183 ceph-mon[33745]: from='client.37634 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi016-75529-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi016-75529-3"}]': finished 2024-03-26T04:20:20.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:19 smithi183 ceph-mon[33745]: from='client.25291 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi016-74362-10"}]': finished 2024-03-26T04:20:20.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:19 smithi183 ceph-mon[33745]: from='client.35360 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi016-74841-12"}]': finished 2024-03-26T04:20:20.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:19 smithi183 ceph-mon[33745]: from='client.25306 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi016-74367-10"}]': finished 2024-03-26T04:20:20.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:19 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/4135966355' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi016-74841-12"}]: dispatch 2024-03-26T04:20:20.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:19 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2150502920' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:20.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:19 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/4137739060' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T04:20:20.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:19 smithi183 ceph-mon[33745]: osdmap e95: 8 total, 8 up, 8 in 2024-03-26T04:20:20.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:19 smithi183 ceph-mon[33745]: from='client.35360 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi016-74841-12"}]: dispatch 2024-03-26T04:20:20.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:19 smithi183 ceph-mon[33745]: from='client.40124 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:20.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:19 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:20.075 INFO:tasks.workunit.client.0.smithi016.stdout: mon: 3 daemons, quorum a,b,c (age 6m) 2024-03-26T04:20:20.075 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: mgr: y(active, since 2m), standbys: x 2024-03-26T04:20:20.075 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: osd: 8 osds: 8 up (since 3m), 8 in (since 3m) 2024-03-26T04:20:20.075 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-03-26T04:20:20.076 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-03-26T04:20:20.076 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: rgw: 1 daemon active (1 hosts, 1 zones) 2024-03-26T04:20:20.076 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: 2024-03-26T04:20:20.076 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: data: 2024-03-26T04:20:20.076 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: pools: 26 pools, 772 pgs 2024-03-26T04:20:20.076 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: objects: 230 objects, 582 KiB 2024-03-26T04:20:20.076 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: usage: 243 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:20:20.076 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: pgs: 38.601% pgs unknown 2024-03-26T04:20:20.076 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: 44.301% pgs not active 2024-03-26T04:20:20.076 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: 342 creating+peering 2024-03-26T04:20:20.076 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: 298 unknown 2024-03-26T04:20:20.076 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: 132 active+clean 2024-03-26T04:20:20.076 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: 2024-03-26T04:20:20.076 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: io: 2024-03-26T04:20:20.077 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: client: 1.2 KiB/s rd, 1 op/s rd, 0 op/s wr 2024-03-26T04:20:20.077 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: 2024-03-26T04:20:20.077 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: 2024-03-26T04:20:20.077 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: [ OK ] LibRadosService.StatusFormat (6191 ms) 2024-03-26T04:20:20.077 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: [ RUN ] LibRadosService.Status 2024-03-26T04:20:20.077 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: [ OK ] LibRadosService.Status (20043 ms) 2024-03-26T04:20:20.077 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: [----------] 4 tests from LibRadosService (31306 ms total) 2024-03-26T04:20:20.077 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: 2024-03-26T04:20:20.077 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: [----------] Global test environment tear-down 2024-03-26T04:20:20.077 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: [==========] 4 tests from 1 test suite ran. (31306 ms total) 2024-03-26T04:20:20.077 INFO:tasks.workunit.client.0.smithi016.stdout: api_service: [ PASSED ] 4 tests. 2024-03-26T04:20:20.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:19 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:20:20.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:19 smithi016 ceph-mon[29101]: from='client.37634 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi016-75529-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi016-75529-3"}]': finished 2024-03-26T04:20:20.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:19 smithi016 ceph-mon[29101]: from='client.25291 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi016-74362-10"}]': finished 2024-03-26T04:20:20.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:19 smithi016 ceph-mon[29101]: from='client.35360 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi016-74841-12"}]': finished 2024-03-26T04:20:20.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:19 smithi016 ceph-mon[29101]: from='client.25306 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi016-74367-10"}]': finished 2024-03-26T04:20:20.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:19 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/4135966355' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi016-74841-12"}]: dispatch 2024-03-26T04:20:20.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:19 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2150502920' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:20.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:19 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/4137739060' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T04:20:20.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:19 smithi016 ceph-mon[29101]: osdmap e95: 8 total, 8 up, 8 in 2024-03-26T04:20:20.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:19 smithi016 ceph-mon[29101]: from='client.35360 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi016-74841-12"}]: dispatch 2024-03-26T04:20:20.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:19 smithi016 ceph-mon[29101]: from='client.40124 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:20.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:19 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:20.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:19 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:20:20.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:19 smithi016 ceph-mon[37215]: from='client.37634 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectsManysmithi016-75529-3", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsManysmithi016-75529-3"}]': finished 2024-03-26T04:20:20.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:19 smithi016 ceph-mon[37215]: from='client.25291 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi016-74362-10"}]': finished 2024-03-26T04:20:20.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:19 smithi016 ceph-mon[37215]: from='client.35360 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyEC_smithi016-74841-12"}]': finished 2024-03-26T04:20:20.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:19 smithi016 ceph-mon[37215]: from='client.25306 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi016-74367-10"}]': finished 2024-03-26T04:20:20.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:19 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/4135966355' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi016-74841-12"}]: dispatch 2024-03-26T04:20:20.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:19 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2150502920' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:20.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:19 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/4137739060' entity='client.admin' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T04:20:20.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:19 smithi016 ceph-mon[37215]: osdmap e95: 8 total, 8 up, 8 in 2024-03-26T04:20:20.165 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:19 smithi016 ceph-mon[37215]: from='client.35360 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi016-74841-12"}]: dispatch 2024-03-26T04:20:20.165 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:19 smithi016 ceph-mon[37215]: from='client.40124 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:20.165 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:19 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:21.234 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:20 smithi016 ceph-mon[29101]: pgmap v102: 592 pgs: 32 creating+peering, 456 active+clean, 104 unknown; 145 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 7.0 KiB/s wr, 5 op/s 2024-03-26T04:20:21.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:20 smithi016 ceph-mon[29101]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T04:20:21.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:20 smithi016 ceph-mon[29101]: from='client.35360 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi016-74841-12"}]': finished 2024-03-26T04:20:21.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:20 smithi016 ceph-mon[29101]: from='client.40124 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:21.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:20 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/326702420' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi016-74264-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:21.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:20 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/304958776' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:21.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:20 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:21.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:20 smithi016 ceph-mon[29101]: osdmap e96: 8 total, 8 up, 8 in 2024-03-26T04:20:21.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:20 smithi016 ceph-mon[29101]: Health check update: 9 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:20:21.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:20 smithi016 ceph-mon[29101]: from='client.31453 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi016-74264-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:21.235 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:20 smithi016 ceph-mon[29101]: from='client.30670 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:21.236 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:20 smithi016 ceph-mon[37215]: pgmap v102: 592 pgs: 32 creating+peering, 456 active+clean, 104 unknown; 145 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 7.0 KiB/s wr, 5 op/s 2024-03-26T04:20:21.236 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:20 smithi016 ceph-mon[37215]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T04:20:21.236 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:20 smithi016 ceph-mon[37215]: from='client.35360 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi016-74841-12"}]': finished 2024-03-26T04:20:21.236 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:20 smithi016 ceph-mon[37215]: from='client.40124 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:21.236 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:20 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/326702420' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi016-74264-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:21.236 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:20 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/304958776' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:21.236 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:20 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:21.236 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:20 smithi016 ceph-mon[37215]: osdmap e96: 8 total, 8 up, 8 in 2024-03-26T04:20:21.236 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:20 smithi016 ceph-mon[37215]: Health check update: 9 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:20:21.236 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:20 smithi016 ceph-mon[37215]: from='client.31453 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi016-74264-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:21.236 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:20 smithi016 ceph-mon[37215]: from='client.30670 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:21.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:20 smithi183 ceph-mon[33745]: pgmap v102: 592 pgs: 32 creating+peering, 456 active+clean, 104 unknown; 145 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 7.0 KiB/s wr, 5 op/s 2024-03-26T04:20:21.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:20 smithi183 ceph-mon[33745]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "service dump"}]: dispatch 2024-03-26T04:20:21.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:20 smithi183 ceph-mon[33745]: from='client.35360 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyEC_smithi016-74841-12"}]': finished 2024-03-26T04:20:21.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:20 smithi183 ceph-mon[33745]: from='client.40124 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:21.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:20 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/326702420' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi016-74264-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:21.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:20 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/304958776' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:21.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:20 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:21.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:20 smithi183 ceph-mon[33745]: osdmap e96: 8 total, 8 up, 8 in 2024-03-26T04:20:21.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:20 smithi183 ceph-mon[33745]: Health check update: 9 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:20:21.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:20 smithi183 ceph-mon[33745]: from='client.31453 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi016-74264-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:21.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:20 smithi183 ceph-mon[33745]: from='client.30670 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:22.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:20:22 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:20:22.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:22 smithi183 ceph-mon[33745]: from='client.31453 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi016-74264-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:22.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:22 smithi183 ceph-mon[33745]: from='client.30670 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:22.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:22 smithi183 ceph-mon[33745]: osdmap e97: 8 total, 8 up, 8 in 2024-03-26T04:20:22.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:22 smithi183 ceph-mon[33745]: pgmap v105: 688 pgs: 32 creating+peering, 456 active+clean, 200 unknown; 145 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 4.5 KiB/s wr, 1 op/s 2024-03-26T04:20:22.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:22 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:22.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:22 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:22.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:22 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:22.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:22 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:20:22.293 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:22 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:20:22.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:22 smithi016 ceph-mon[37215]: from='client.31453 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi016-74264-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:22.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:22 smithi016 ceph-mon[37215]: from='client.30670 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:22.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:22 smithi016 ceph-mon[37215]: osdmap e97: 8 total, 8 up, 8 in 2024-03-26T04:20:22.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:22 smithi016 ceph-mon[37215]: pgmap v105: 688 pgs: 32 creating+peering, 456 active+clean, 200 unknown; 145 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 4.5 KiB/s wr, 1 op/s 2024-03-26T04:20:22.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:22 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:22.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:22 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:22.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:22 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:22.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:22 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:20:22.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:22 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:20:22.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:22 smithi016 ceph-mon[29101]: from='client.31453 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTrip3_smithi016-74264-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:22.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:22 smithi016 ceph-mon[29101]: from='client.30670 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:22.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:22 smithi016 ceph-mon[29101]: osdmap e97: 8 total, 8 up, 8 in 2024-03-26T04:20:22.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:22 smithi016 ceph-mon[29101]: pgmap v105: 688 pgs: 32 creating+peering, 456 active+clean, 200 unknown; 145 MiB data, 766 MiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 4.5 KiB/s wr, 1 op/s 2024-03-26T04:20:22.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:22 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:22.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:22 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:22.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:22 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:22.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:22 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:20:22.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:22 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:20:23.504 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:23 smithi016 ceph-mon[29101]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T04:20:23.505 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:23 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:23.505 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:23 smithi016 ceph-mon[29101]: osdmap e98: 8 total, 8 up, 8 in 2024-03-26T04:20:23.505 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:23 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:23.505 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:23 smithi016 ceph-mon[37215]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T04:20:23.505 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:23 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:23.505 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:23 smithi016 ceph-mon[37215]: osdmap e98: 8 total, 8 up, 8 in 2024-03-26T04:20:23.505 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:23 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:23.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:23 smithi183 ceph-mon[33745]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T04:20:23.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:23 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:23.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:23 smithi183 ceph-mon[33745]: osdmap e98: 8 total, 8 up, 8 in 2024-03-26T04:20:23.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:23 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:24.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:20:24.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[37215]: pgmap v107: 656 pgs: 91 creating+activating, 49 creating+peering, 483 active+clean, 4 active+clean+snaptrim_wait, 29 unknown; 145 MiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.6 KiB/s rd, 2 op/s 2024-03-26T04:20:24.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:24.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[37215]: osdmap e99: 8 total, 8 up, 8 in 2024-03-26T04:20:24.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/4261059399' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi016-75529-3"}]: dispatch 2024-03-26T04:20:24.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:24.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3471836089' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi016-74264-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:24.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:24.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[37215]: from='client.37634 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi016-75529-3"}]: dispatch 2024-03-26T04:20:24.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[37215]: from='client.34705 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi016-74264-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:24.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:20:24.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:20:24.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:24.417 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:24.417 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[37215]: from='client.37634 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi016-75529-3"}]': finished 2024-03-26T04:20:24.417 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[37215]: from='client.34705 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi016-74264-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:24.417 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[37215]: osdmap e100: 8 total, 8 up, 8 in 2024-03-26T04:20:24.417 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3045560109' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi016-74285-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:24.417 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3334105217' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:24.417 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/4261059399' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi016-75529-3"}]: dispatch 2024-03-26T04:20:24.417 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[37215]: from='client.37634 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi016-75529-3"}]: dispatch 2024-03-26T04:20:24.417 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[37215]: from='client.35797 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi016-74285-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:24.417 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[37215]: from='client.35845 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:24.434 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:20:24.434 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[29101]: pgmap v107: 656 pgs: 91 creating+activating, 49 creating+peering, 483 active+clean, 4 active+clean+snaptrim_wait, 29 unknown; 145 MiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.6 KiB/s rd, 2 op/s 2024-03-26T04:20:24.434 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:24.434 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[29101]: osdmap e99: 8 total, 8 up, 8 in 2024-03-26T04:20:24.434 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/4261059399' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi016-75529-3"}]: dispatch 2024-03-26T04:20:24.434 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:24.434 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3471836089' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi016-74264-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:24.434 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:24.434 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[29101]: from='client.37634 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi016-75529-3"}]: dispatch 2024-03-26T04:20:24.434 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[29101]: from='client.34705 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi016-74264-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:24.434 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:20:24.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:20:24.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:24.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:24.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[29101]: from='client.37634 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi016-75529-3"}]': finished 2024-03-26T04:20:24.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[29101]: from='client.34705 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi016-74264-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:24.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[29101]: osdmap e100: 8 total, 8 up, 8 in 2024-03-26T04:20:24.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3045560109' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi016-74285-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:24.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3334105217' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:24.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/4261059399' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi016-75529-3"}]: dispatch 2024-03-26T04:20:24.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[29101]: from='client.37634 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi016-75529-3"}]: dispatch 2024-03-26T04:20:24.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[29101]: from='client.35797 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi016-74285-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:24.435 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:24 smithi016 ceph-mon[29101]: from='client.35845 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:24.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:24 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:20:24.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:24 smithi183 ceph-mon[33745]: pgmap v107: 656 pgs: 91 creating+activating, 49 creating+peering, 483 active+clean, 4 active+clean+snaptrim_wait, 29 unknown; 145 MiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.6 KiB/s rd, 2 op/s 2024-03-26T04:20:24.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:24 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:24.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:24 smithi183 ceph-mon[33745]: osdmap e99: 8 total, 8 up, 8 in 2024-03-26T04:20:24.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:24 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/4261059399' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi016-75529-3"}]: dispatch 2024-03-26T04:20:24.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:24 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:24.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:24 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3471836089' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi016-74264-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:24.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:24 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:24.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:24 smithi183 ceph-mon[33745]: from='client.37634 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi016-75529-3"}]: dispatch 2024-03-26T04:20:24.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:24 smithi183 ceph-mon[33745]: from='client.34705 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi016-74264-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:24.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:24 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:20:24.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:24 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:20:24.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:24 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:24.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:24 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:24.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:24 smithi183 ceph-mon[33745]: from='client.37634 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectsManysmithi016-75529-3"}]': finished 2024-03-26T04:20:24.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:24 smithi183 ceph-mon[33745]: from='client.34705 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppend_smithi016-74264-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:24.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:24 smithi183 ceph-mon[33745]: osdmap e100: 8 total, 8 up, 8 in 2024-03-26T04:20:24.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:24 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3045560109' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi016-74285-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:24.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:24 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3334105217' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:24.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:24 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/4261059399' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi016-75529-3"}]: dispatch 2024-03-26T04:20:24.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:24 smithi183 ceph-mon[33745]: from='client.37634 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi016-75529-3"}]: dispatch 2024-03-26T04:20:24.544 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:24 smithi183 ceph-mon[33745]: from='client.35797 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi016-74285-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:24.544 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:24 smithi183 ceph-mon[33745]: from='client.35845 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:25.450 INFO:tasks.workunit.client.0.smithi016.stdout: ec_list: Running main() from gmock_main.cc 2024-03-26T04:20:25.451 INFO:tasks.workunit.client.0.smithi016.stdout: ec_list: [==========] Running 3 tests from 1 test suite. 2024-03-26T04:20:25.451 INFO:tasks.workunit.client.0.smithi016.stdout: ec_list: [----------] Global test environment set-up. 2024-03-26T04:20:25.451 INFO:tasks.workunit.client.0.smithi016.stdout: ec_list: [----------] 3 tests from NeoradosECList 2024-03-26T04:20:25.451 INFO:tasks.workunit.client.0.smithi016.stdout: ec_list: [ RUN ] NeoradosECList.ListObjects 2024-03-26T04:20:25.451 INFO:tasks.workunit.client.0.smithi016.stdout: ec_list: [ OK ] NeoradosECList.ListObjects (18032 ms) 2024-03-26T04:20:25.451 INFO:tasks.workunit.client.0.smithi016.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsNS 2024-03-26T04:20:25.451 INFO:tasks.workunit.client.0.smithi016.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsNS (8389 ms) 2024-03-26T04:20:25.451 INFO:tasks.workunit.client.0.smithi016.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsMany 2024-03-26T04:20:25.451 INFO:tasks.workunit.client.0.smithi016.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsMany (10087 ms) 2024-03-26T04:20:25.451 INFO:tasks.workunit.client.0.smithi016.stdout: ec_list: [----------] 3 tests from NeoradosECList (36509 ms total) 2024-03-26T04:20:25.451 INFO:tasks.workunit.client.0.smithi016.stdout: ec_list: 2024-03-26T04:20:25.451 INFO:tasks.workunit.client.0.smithi016.stdout: ec_list: [----------] Global test environment tear-down 2024-03-26T04:20:25.452 INFO:tasks.workunit.client.0.smithi016.stdout: ec_list: [==========] 3 tests from 1 test suite ran. (36509 ms total) 2024-03-26T04:20:25.452 INFO:tasks.workunit.client.0.smithi016.stdout: ec_list: [ PASSED ] 3 tests. 2024-03-26T04:20:26.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:25 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:20:26.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:25 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:20:26.166 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:25 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:20:27.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:26 smithi183 ceph-mon[33745]: pgmap v110: 648 pgs: 53 creating+activating, 8 creating+peering, 423 active+clean, 4 active+clean+snaptrim_wait, 160 unknown; 145 MiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 2 op/s 2024-03-26T04:20:27.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:26 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:27.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:26 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:20:27.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:26 smithi183 ceph-mon[33745]: from='client.37634 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi016-75529-3"}]': finished 2024-03-26T04:20:27.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:26 smithi183 ceph-mon[33745]: from='client.35797 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi016-74285-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:27.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:26 smithi183 ceph-mon[33745]: from='client.35845 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:27.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:26 smithi183 ceph-mon[33745]: osdmap e101: 8 total, 8 up, 8 in 2024-03-26T04:20:27.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:26 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3457665461' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:27.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:26 smithi183 ceph-mon[33745]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:20:27.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:26 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-7", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:20:27.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:26 smithi183 ceph-mon[33745]: osdmap e102: 8 total, 8 up, 8 in 2024-03-26T04:20:27.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:26 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-7"}]: dispatch 2024-03-26T04:20:27.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:26 smithi183 ceph-mon[33745]: from='client.37102 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:27.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:26 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-7"}]: dispatch 2024-03-26T04:20:27.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:26 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:27.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:26 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:20:27.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:26 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:20:27.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:26 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3878746505' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi016-74781-4"}]: dispatch 2024-03-26T04:20:27.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:26 smithi183 ceph-mon[33745]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi016-74781-4"}]: dispatch 2024-03-26T04:20:27.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:26 smithi016 ceph-mon[29101]: pgmap v110: 648 pgs: 53 creating+activating, 8 creating+peering, 423 active+clean, 4 active+clean+snaptrim_wait, 160 unknown; 145 MiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 2 op/s 2024-03-26T04:20:27.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:26 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:27.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:26 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:20:27.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:26 smithi016 ceph-mon[29101]: from='client.37634 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi016-75529-3"}]': finished 2024-03-26T04:20:27.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:26 smithi016 ceph-mon[29101]: from='client.35797 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi016-74285-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:27.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:26 smithi016 ceph-mon[29101]: from='client.35845 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:27.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:26 smithi016 ceph-mon[29101]: osdmap e101: 8 total, 8 up, 8 in 2024-03-26T04:20:27.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:26 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3457665461' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:27.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:26 smithi016 ceph-mon[29101]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:20:27.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:26 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-7", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:20:27.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:26 smithi016 ceph-mon[29101]: osdmap e102: 8 total, 8 up, 8 in 2024-03-26T04:20:27.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:26 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-7"}]: dispatch 2024-03-26T04:20:27.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:26 smithi016 ceph-mon[29101]: from='client.37102 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:27.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:26 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-7"}]: dispatch 2024-03-26T04:20:27.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:26 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:27.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:26 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:20:27.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:26 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:20:27.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:26 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3878746505' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi016-74781-4"}]: dispatch 2024-03-26T04:20:27.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:26 smithi016 ceph-mon[29101]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi016-74781-4"}]: dispatch 2024-03-26T04:20:27.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:26 smithi016 ceph-mon[37215]: pgmap v110: 648 pgs: 53 creating+activating, 8 creating+peering, 423 active+clean, 4 active+clean+snaptrim_wait, 160 unknown; 145 MiB data, 772 MiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 2 op/s 2024-03-26T04:20:27.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:26 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:27.165 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:26 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-7", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:20:27.165 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:26 smithi016 ceph-mon[37215]: from='client.37634 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectsManysmithi016-75529-3"}]': finished 2024-03-26T04:20:27.165 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:26 smithi016 ceph-mon[37215]: from='client.35797 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi016-74285-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:27.165 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:26 smithi016 ceph-mon[37215]: from='client.35845 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:27.165 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:26 smithi016 ceph-mon[37215]: osdmap e101: 8 total, 8 up, 8 in 2024-03-26T04:20:27.165 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:26 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3457665461' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:27.165 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:26 smithi016 ceph-mon[37215]: Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:20:27.165 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:26 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-7", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:20:27.165 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:26 smithi016 ceph-mon[37215]: osdmap e102: 8 total, 8 up, 8 in 2024-03-26T04:20:27.165 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:26 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-7"}]: dispatch 2024-03-26T04:20:27.165 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:26 smithi016 ceph-mon[37215]: from='client.37102 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:27.165 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:26 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-7"}]: dispatch 2024-03-26T04:20:27.165 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:26 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:27.165 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:26 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:20:27.165 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:26 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:20:27.166 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:26 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3878746505' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi016-74781-4"}]: dispatch 2024-03-26T04:20:27.166 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:26 smithi016 ceph-mon[37215]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi016-74781-4"}]: dispatch 2024-03-26T04:20:28.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:27 smithi016 ceph-mon[29101]: from='client.37102 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:28.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:27 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-7"}]': finished 2024-03-26T04:20:28.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:27 smithi016 ceph-mon[29101]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi016-74781-4"}]': finished 2024-03-26T04:20:28.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:27 smithi016 ceph-mon[29101]: osdmap e103: 8 total, 8 up, 8 in 2024-03-26T04:20:28.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:27 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3878746505' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi016-74781-4", "tierpool": "test-rados-api-smithi016-74781-6"}]: dispatch 2024-03-26T04:20:28.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:27 smithi016 ceph-mon[29101]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi016-74781-4", "tierpool": "test-rados-api-smithi016-74781-6"}]: dispatch 2024-03-26T04:20:28.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:27 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1510515131' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi016-74264-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:28.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:27 smithi016 ceph-mon[29101]: from='client.38491 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi016-74264-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:28.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:27 smithi016 ceph-mon[29101]: pgmap v114: 680 pgs: 6 creating+peering, 28 creating+activating, 446 active+clean, 4 active+clean+snaptrim_wait, 196 unknown; 145 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 268 B/s wr, 0 op/s 2024-03-26T04:20:28.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:27 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:20:28.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:27 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:28.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:27 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:20:28.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:27 smithi016 ceph-mon[37215]: from='client.37102 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:28.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:27 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-7"}]': finished 2024-03-26T04:20:28.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:27 smithi016 ceph-mon[37215]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi016-74781-4"}]': finished 2024-03-26T04:20:28.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:27 smithi016 ceph-mon[37215]: osdmap e103: 8 total, 8 up, 8 in 2024-03-26T04:20:28.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:27 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3878746505' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi016-74781-4", "tierpool": "test-rados-api-smithi016-74781-6"}]: dispatch 2024-03-26T04:20:28.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:27 smithi016 ceph-mon[37215]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi016-74781-4", "tierpool": "test-rados-api-smithi016-74781-6"}]: dispatch 2024-03-26T04:20:28.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:27 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1510515131' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi016-74264-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:28.165 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:27 smithi016 ceph-mon[37215]: from='client.38491 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi016-74264-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:28.165 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:27 smithi016 ceph-mon[37215]: pgmap v114: 680 pgs: 6 creating+peering, 28 creating+activating, 446 active+clean, 4 active+clean+snaptrim_wait, 196 unknown; 145 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 268 B/s wr, 0 op/s 2024-03-26T04:20:28.165 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:27 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:20:28.165 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:27 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:28.165 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:27 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:20:28.236 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:27 smithi183 ceph-mon[33745]: from='client.37102 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74992-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:28.236 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:27 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-7"}]': finished 2024-03-26T04:20:28.236 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:27 smithi183 ceph-mon[33745]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi016-74781-4"}]': finished 2024-03-26T04:20:28.237 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:27 smithi183 ceph-mon[33745]: osdmap e103: 8 total, 8 up, 8 in 2024-03-26T04:20:28.237 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:27 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3878746505' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi016-74781-4", "tierpool": "test-rados-api-smithi016-74781-6"}]: dispatch 2024-03-26T04:20:28.237 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:27 smithi183 ceph-mon[33745]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi016-74781-4", "tierpool": "test-rados-api-smithi016-74781-6"}]: dispatch 2024-03-26T04:20:28.237 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:27 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1510515131' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi016-74264-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:28.237 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:27 smithi183 ceph-mon[33745]: from='client.38491 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi016-74264-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:28.237 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:27 smithi183 ceph-mon[33745]: pgmap v114: 680 pgs: 6 creating+peering, 28 creating+activating, 446 active+clean, 4 active+clean+snaptrim_wait, 196 unknown; 145 MiB data, 777 MiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 268 B/s wr, 0 op/s 2024-03-26T04:20:28.237 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:27 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:20:28.237 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:27 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:28.237 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:27 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:20:29.063 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:29 smithi016 ceph-mon[29101]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:20:29.063 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:29 smithi016 ceph-mon[29101]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi016-74781-4", "tierpool": "test-rados-api-smithi016-74781-6"}]': finished 2024-03-26T04:20:29.063 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:29 smithi016 ceph-mon[29101]: from='client.38491 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi016-74264-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:29.063 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:29 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:20:29.063 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:29 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3878746505' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi016-74781-6", "pool2": "test-rados-api-smithi016-74781-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:20:29.064 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:20:28 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:20:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:20:29.094 INFO:tasks.workunit.client.0.smithi016.stdout: write_operations: Running main() from gmock_main.cc 2024-03-26T04:20:29.094 INFO:tasks.workunit.client.0.smithi016.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2024-03-26T04:20:29.094 INFO:tasks.workunit.client.0.smithi016.stdout: write_operations: [----------] Global test environment set-up. 2024-03-26T04:20:29.094 INFO:tasks.workunit.client.0.smithi016.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2024-03-26T04:20:29.094 INFO:tasks.workunit.client.0.smithi016.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2024-03-26T04:20:29.094 INFO:tasks.workunit.client.0.smithi016.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (10764 ms) 2024-03-26T04:20:29.094 INFO:tasks.workunit.client.0.smithi016.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2024-03-26T04:20:29.094 INFO:tasks.workunit.client.0.smithi016.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (4094 ms) 2024-03-26T04:20:29.095 INFO:tasks.workunit.client.0.smithi016.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2024-03-26T04:20:29.095 INFO:tasks.workunit.client.0.smithi016.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (5444 ms) 2024-03-26T04:20:29.095 INFO:tasks.workunit.client.0.smithi016.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2024-03-26T04:20:29.095 INFO:tasks.workunit.client.0.smithi016.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (6094 ms) 2024-03-26T04:20:29.095 INFO:tasks.workunit.client.0.smithi016.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2024-03-26T04:20:29.095 INFO:tasks.workunit.client.0.smithi016.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (3250 ms) 2024-03-26T04:20:29.095 INFO:tasks.workunit.client.0.smithi016.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2024-03-26T04:20:29.095 INFO:tasks.workunit.client.0.smithi016.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (3391 ms) 2024-03-26T04:20:29.095 INFO:tasks.workunit.client.0.smithi016.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2024-03-26T04:20:29.095 INFO:tasks.workunit.client.0.smithi016.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (7055 ms) 2024-03-26T04:20:29.095 INFO:tasks.workunit.client.0.smithi016.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (40093 ms total) 2024-03-26T04:20:29.095 INFO:tasks.workunit.client.0.smithi016.stdout: write_operations: 2024-03-26T04:20:29.095 INFO:tasks.workunit.client.0.smithi016.stdout: write_operations: [----------] Global test environment tear-down 2024-03-26T04:20:29.096 INFO:tasks.workunit.client.0.smithi016.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (40093 ms total) 2024-03-26T04:20:29.096 INFO:tasks.workunit.client.0.smithi016.stdout: write_operations: [ PASSED ] 7 tests. 2024-03-26T04:20:29.320 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:29 smithi183 ceph-mon[33745]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:20:29.321 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:29 smithi183 ceph-mon[33745]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi016-74781-4", "tierpool": "test-rados-api-smithi016-74781-6"}]': finished 2024-03-26T04:20:29.321 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:29 smithi183 ceph-mon[33745]: from='client.38491 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi016-74264-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:29.321 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:29 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:20:29.321 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:29 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3878746505' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi016-74781-6", "pool2": "test-rados-api-smithi016-74781-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:20:29.321 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:29 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-7"}]: dispatch 2024-03-26T04:20:29.321 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:29 smithi183 ceph-mon[33745]: osdmap e104: 8 total, 8 up, 8 in 2024-03-26T04:20:29.321 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:29 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1349170912' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi016-74285-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:29.321 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:29 smithi183 ceph-mon[33745]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi016-74781-6", "pool2": "test-rados-api-smithi016-74781-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:20:29.321 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:29 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-7"}]: dispatch 2024-03-26T04:20:29.321 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:29 smithi183 ceph-mon[33745]: from='client.39511 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi016-74285-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:29.321 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:29 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2934028528' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi016-74578-10"}]: dispatch 2024-03-26T04:20:29.321 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:29 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:29.322 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:29 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:20:29.322 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:29 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:20:29.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:29 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-7"}]: dispatch 2024-03-26T04:20:29.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:29 smithi016 ceph-mon[29101]: osdmap e104: 8 total, 8 up, 8 in 2024-03-26T04:20:29.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:29 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1349170912' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi016-74285-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:29.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:29 smithi016 ceph-mon[29101]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi016-74781-6", "pool2": "test-rados-api-smithi016-74781-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:20:29.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:29 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-7"}]: dispatch 2024-03-26T04:20:29.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:29 smithi016 ceph-mon[29101]: from='client.39511 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi016-74285-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:29.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:29 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2934028528' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi016-74578-10"}]: dispatch 2024-03-26T04:20:29.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:29 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:29.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:29 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:20:29.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:29 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:20:29.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:29 smithi016 ceph-mon[37215]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:20:29.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:29 smithi016 ceph-mon[37215]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi016-74781-4", "tierpool": "test-rados-api-smithi016-74781-6"}]': finished 2024-03-26T04:20:29.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:29 smithi016 ceph-mon[37215]: from='client.38491 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTest_smithi016-74264-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:29.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:29 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:20:29.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:29 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3878746505' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi016-74781-6", "pool2": "test-rados-api-smithi016-74781-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:20:29.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:29 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-7"}]: dispatch 2024-03-26T04:20:29.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:29 smithi016 ceph-mon[37215]: osdmap e104: 8 total, 8 up, 8 in 2024-03-26T04:20:29.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:29 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1349170912' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi016-74285-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:29.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:29 smithi016 ceph-mon[37215]: from='client.35288 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi016-74781-6", "pool2": "test-rados-api-smithi016-74781-6", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:20:29.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:29 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-7"}]: dispatch 2024-03-26T04:20:29.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:29 smithi016 ceph-mon[37215]: from='client.39511 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi016-74285-5","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:29.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:29 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2934028528' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi016-74578-10"}]: dispatch 2024-03-26T04:20:29.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:29 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:29.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:29 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:20:29.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:29 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:20:30.099 INFO:tasks.workunit.client.0.smithi016.stdout:i_watch_notify_pp: flushed 2024-03-26T04:20:30.099 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3007 ms) 2024-03-26T04:20:30.099 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-03-26T04:20:30.100 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: trying... 2024-03-26T04:20:30.100 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: handle_notify cookie 94258755786048 notify_id 399431958528 notifier_gid 35288 2024-03-26T04:20:30.100 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: timed out 2024-03-26T04:20:30.100 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: flushing 2024-03-26T04:20:30.100 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: flushed 2024-03-26T04:20:30.100 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3015 ms) 2024-03-26T04:20:30.100 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-03-26T04:20:30.100 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: List watches 2024-03-26T04:20:30.100 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: notify2 2024-03-26T04:20:30.100 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: handle_notify cookie 94258755786048 notify_id 412316860423 notifier_gid 35288 2024-03-26T04:20:30.100 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: notify2 done 2024-03-26T04:20:30.100 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: watch_check 2024-03-26T04:20:30.100 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: unwatch2 2024-03-26T04:20:30.100 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: flushing 2024-03-26T04:20:30.101 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: done 2024-03-26T04:20:30.101 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3094 ms) 2024-03-26T04:20:30.101 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-03-26T04:20:30.101 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: List watches 2024-03-26T04:20:30.101 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: notify2 2024-03-26T04:20:30.101 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: handle_notify cookie 94258755786048 notify_id 425201762312 notifier_gid 35288 2024-03-26T04:20:30.101 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: notify2 done 2024-03-26T04:20:30.101 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: watch_check 2024-03-26T04:20:30.101 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: unwatch2 2024-03-26T04:20:30.101 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: flushing 2024-03-26T04:20:30.101 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: done 2024-03-26T04:20:30.101 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3025 ms) 2024-03-26T04:20:30.101 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (18284 ms total) 2024-03-26T04:20:30.101 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: 2024-03-26T04:20:30.102 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-03-26T04:20:30.102 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (41387 ms total) 2024-03-26T04:20:30.102 INFO:tasks.workunit.client.0.smithi016.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-03-26T04:20:30.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:30 smithi183 ceph-mon[33745]: pgmap v116: 616 pgs: 32 creating+peering, 520 active+clean, 64 unknown; 145 MiB data, 778 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-03-26T04:20:30.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:30 smithi183 ceph-mon[33745]: from='client.40450 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi016-74578-10"}]: dispatch 2024-03-26T04:20:30.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:30 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2934028528' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi016-74578-10"}]: dispatch 2024-03-26T04:20:30.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:30 smithi183 ceph-mon[33745]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi016-74781-6", "pool2": "test-rados-api-smithi016-74781-6", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T04:20:30.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:30 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-7"}]': finished 2024-03-26T04:20:30.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:30 smithi183 ceph-mon[33745]: from='client.39511 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi016-74285-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:30.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:30 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:20:30.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:30 smithi183 ceph-mon[33745]: osdmap e105: 8 total, 8 up, 8 in 2024-03-26T04:20:30.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:30 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1955025999' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:30.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:30 smithi183 ceph-mon[33745]: from='client.40450 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi016-74578-10"}]: dispatch 2024-03-26T04:20:30.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:30 smithi183 ceph-mon[33745]: from='client.40444 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:30.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:30 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2934028528' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi016-74578-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:30.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:30 smithi183 ceph-mon[33745]: from='client.40450 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi016-74578-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:30.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:30 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:30.917 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:30 smithi016 ceph-mon[37215]: pgmap v116: 616 pgs: 32 creating+peering, 520 active+clean, 64 unknown; 145 MiB data, 778 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-03-26T04:20:30.917 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:30 smithi016 ceph-mon[37215]: from='client.40450 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi016-74578-10"}]: dispatch 2024-03-26T04:20:30.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:30 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2934028528' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi016-74578-10"}]: dispatch 2024-03-26T04:20:30.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:30 smithi016 ceph-mon[37215]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi016-74781-6", "pool2": "test-rados-api-smithi016-74781-6", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T04:20:30.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:30 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-7"}]': finished 2024-03-26T04:20:30.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:30 smithi016 ceph-mon[37215]: from='client.39511 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi016-74285-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:30.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:30 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:20:30.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:30 smithi016 ceph-mon[37215]: osdmap e105: 8 total, 8 up, 8 in 2024-03-26T04:20:30.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:30 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1955025999' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:30.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:30 smithi016 ceph-mon[37215]: from='client.40450 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi016-74578-10"}]: dispatch 2024-03-26T04:20:30.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:30 smithi016 ceph-mon[37215]: from='client.40444 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:30.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:30 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2934028528' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi016-74578-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:30.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:30 smithi016 ceph-mon[37215]: from='client.40450 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi016-74578-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:30.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:30 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:30.919 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:30 smithi016 ceph-mon[29101]: pgmap v116: 616 pgs: 32 creating+peering, 520 active+clean, 64 unknown; 145 MiB data, 778 MiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-03-26T04:20:30.919 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:30 smithi016 ceph-mon[29101]: from='client.40450 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi016-74578-10"}]: dispatch 2024-03-26T04:20:30.919 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:30 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2934028528' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi016-74578-10"}]: dispatch 2024-03-26T04:20:30.919 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:30 smithi016 ceph-mon[29101]: from='client.35288 ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "test-rados-api-smithi016-74781-6", "pool2": "test-rados-api-smithi016-74781-6", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T04:20:30.919 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:30 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-7"}]': finished 2024-03-26T04:20:30.919 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:30 smithi016 ceph-mon[29101]: from='client.39511 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWritePP_smithi016-74285-5","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:30.919 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:30 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:20:30.919 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:30 smithi016 ceph-mon[29101]: osdmap e105: 8 total, 8 up, 8 in 2024-03-26T04:20:30.919 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:30 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1955025999' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:30.920 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:30 smithi016 ceph-mon[29101]: from='client.40450 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi016-74578-10"}]: dispatch 2024-03-26T04:20:30.920 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:30 smithi016 ceph-mon[29101]: from='client.40444 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:30.920 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:30 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2934028528' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi016-74578-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:30.920 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:30 smithi016 ceph-mon[29101]: from='client.40450 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi016-74578-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:30.920 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:30 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:31.881 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-03-26T04:20:31.881 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-03-26T04:20:31.882 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-03-26T04:20:31.882 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-03-26T04:20:31.882 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-03-26T04:20:31.882 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-03-26T04:20:31.882 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-03-26T04:20:31.882 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_write_operations: 2024-03-26T04:20:31.882 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-03-26T04:20:31.882 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-03-26T04:20:31.882 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (12915 ms) 2024-03-26T04:20:31.882 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-03-26T04:20:31.882 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (3216 ms) 2024-03-26T04:20:31.882 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-03-26T04:20:31.882 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (4387 ms) 2024-03-26T04:20:31.882 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-03-26T04:20:31.883 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (3589 ms) 2024-03-26T04:20:31.883 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-03-26T04:20:31.883 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (5019 ms) 2024-03-26T04:20:31.883 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-03-26T04:20:31.883 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (6309 ms) 2024-03-26T04:20:31.883 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-03-26T04:20:31.883 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (7665 ms) 2024-03-26T04:20:31.883 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (43100 ms total) 2024-03-26T04:20:31.883 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_write_operations: 2024-03-26T04:20:31.883 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-03-26T04:20:31.883 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (43101 ms total) 2024-03-26T04:20:31.883 INFO:tasks.workunit.client.0.smithi016.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-03-26T04:20:31.917 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:31 smithi016 ceph-mon[37215]: from='client.40444 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:31.917 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:31 smithi016 ceph-mon[37215]: from='client.40450 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi016-74578-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:20:31.917 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:31 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2934028528' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi016-74578-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi016-74578-10"}]: dispatch 2024-03-26T04:20:31.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:31 smithi016 ceph-mon[37215]: osdmap e106: 8 total, 8 up, 8 in 2024-03-26T04:20:31.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:31 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2253561028' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi016-74264-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:31.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:31 smithi016 ceph-mon[37215]: from='client.40450 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi016-74578-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi016-74578-10"}]: dispatch 2024-03-26T04:20:31.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:31 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:31.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:31 smithi016 ceph-mon[37215]: from='client.41686 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi016-74264-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:31.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:31 smithi016 ceph-mon[37215]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:20:31.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:31 smithi016 ceph-mon[37215]: from='client.41686 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi016-74264-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:31.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:31 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:31.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:31 smithi016 ceph-mon[37215]: osdmap e107: 8 total, 8 up, 8 in 2024-03-26T04:20:31.918 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:31 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:31.919 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:31 smithi016 ceph-mon[29101]: from='client.40444 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:31.919 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:31 smithi016 ceph-mon[29101]: from='client.40450 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi016-74578-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:20:31.919 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:31 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2934028528' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi016-74578-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi016-74578-10"}]: dispatch 2024-03-26T04:20:31.919 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:31 smithi016 ceph-mon[29101]: osdmap e106: 8 total, 8 up, 8 in 2024-03-26T04:20:31.919 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:31 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2253561028' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi016-74264-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:31.919 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:31 smithi016 ceph-mon[29101]: from='client.40450 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi016-74578-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi016-74578-10"}]: dispatch 2024-03-26T04:20:31.919 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:31 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:31.919 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:31 smithi016 ceph-mon[29101]: from='client.41686 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi016-74264-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:31.919 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:31 smithi016 ceph-mon[29101]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:20:31.920 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:31 smithi016 ceph-mon[29101]: from='client.41686 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi016-74264-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:31.920 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:31 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:31.920 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:31 smithi016 ceph-mon[29101]: osdmap e107: 8 total, 8 up, 8 in 2024-03-26T04:20:31.920 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:31 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:32.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:31 smithi183 ceph-mon[33745]: from='client.40444 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:32.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:31 smithi183 ceph-mon[33745]: from='client.40450 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsEC_smithi016-74578-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:20:32.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:31 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2934028528' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi016-74578-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi016-74578-10"}]: dispatch 2024-03-26T04:20:32.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:31 smithi183 ceph-mon[33745]: osdmap e106: 8 total, 8 up, 8 in 2024-03-26T04:20:32.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:31 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2253561028' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi016-74264-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:32.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:31 smithi183 ceph-mon[33745]: from='client.40450 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi016-74578-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi016-74578-10"}]: dispatch 2024-03-26T04:20:32.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:31 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:32.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:31 smithi183 ceph-mon[33745]: from='client.41686 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi016-74264-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:32.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:31 smithi183 ceph-mon[33745]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:20:32.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:31 smithi183 ceph-mon[33745]: from='client.41686 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTrip_smithi016-74264-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:32.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:31 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:32.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:31 smithi183 ceph-mon[33745]: osdmap e107: 8 total, 8 up, 8 in 2024-03-26T04:20:32.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:31 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:32.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:20:32 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:20:33.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[29101]: pgmap v120: 644 pgs: 420 active+clean, 224 unknown; 145 MiB data, 778 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:20:33.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.0"}]: dispatch 2024-03-26T04:20:33.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[29101]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.0"}]: dispatch 2024-03-26T04:20:33.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.1"}]: dispatch 2024-03-26T04:20:33.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[29101]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.1"}]: dispatch 2024-03-26T04:20:33.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.2"}]: dispatch 2024-03-26T04:20:33.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[29101]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.2"}]: dispatch 2024-03-26T04:20:33.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.3"}]: dispatch 2024-03-26T04:20:33.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[29101]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.3"}]: dispatch 2024-03-26T04:20:33.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.4"}]: dispatch 2024-03-26T04:20:33.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[29101]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.4"}]: dispatch 2024-03-26T04:20:33.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.5"}]: dispatch 2024-03-26T04:20:33.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[29101]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.5"}]: dispatch 2024-03-26T04:20:33.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.6"}]: dispatch 2024-03-26T04:20:33.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[29101]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.6"}]: dispatch 2024-03-26T04:20:33.416 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.7"}]: dispatch 2024-03-26T04:20:33.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[29101]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.7"}]: dispatch 2024-03-26T04:20:33.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.8"}]: dispatch 2024-03-26T04:20:33.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[29101]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.8"}]: dispatch 2024-03-26T04:20:33.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.9"}]: dispatch 2024-03-26T04:20:33.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[29101]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.9"}]: dispatch 2024-03-26T04:20:33.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:33.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[29101]: from='client.40450 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi016-74578-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi016-74578-10"}]': finished 2024-03-26T04:20:33.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:33.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[29101]: osdmap e108: 8 total, 8 up, 8 in 2024-03-26T04:20:33.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1237666305' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi016-74285-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:33.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[29101]: from='client.43087 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi016-74285-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:33.417 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:20:33.418 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:33.418 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[37215]: pgmap v120: 644 pgs: 420 active+clean, 224 unknown; 145 MiB data, 778 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:20:33.418 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.0"}]: dispatch 2024-03-26T04:20:33.418 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[37215]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.0"}]: dispatch 2024-03-26T04:20:33.418 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.1"}]: dispatch 2024-03-26T04:20:33.418 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[37215]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.1"}]: dispatch 2024-03-26T04:20:33.418 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.2"}]: dispatch 2024-03-26T04:20:33.418 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[37215]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.2"}]: dispatch 2024-03-26T04:20:33.418 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.3"}]: dispatch 2024-03-26T04:20:33.419 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[37215]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.3"}]: dispatch 2024-03-26T04:20:33.419 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.4"}]: dispatch 2024-03-26T04:20:33.419 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[37215]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.4"}]: dispatch 2024-03-26T04:20:33.419 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.5"}]: dispatch 2024-03-26T04:20:33.419 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[37215]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.5"}]: dispatch 2024-03-26T04:20:33.419 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.6"}]: dispatch 2024-03-26T04:20:33.419 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[37215]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.6"}]: dispatch 2024-03-26T04:20:33.419 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.7"}]: dispatch 2024-03-26T04:20:33.419 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[37215]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.7"}]: dispatch 2024-03-26T04:20:33.419 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.8"}]: dispatch 2024-03-26T04:20:33.419 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[37215]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.8"}]: dispatch 2024-03-26T04:20:33.419 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.9"}]: dispatch 2024-03-26T04:20:33.420 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[37215]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.9"}]: dispatch 2024-03-26T04:20:33.420 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:33.420 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[37215]: from='client.40450 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi016-74578-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi016-74578-10"}]': finished 2024-03-26T04:20:33.420 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:33.420 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[37215]: osdmap e108: 8 total, 8 up, 8 in 2024-03-26T04:20:33.420 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1237666305' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi016-74285-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:33.420 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[37215]: from='client.43087 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi016-74285-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:33.420 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:20:33.420 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:33 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:33.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:33 smithi183 ceph-mon[33745]: pgmap v120: 644 pgs: 420 active+clean, 224 unknown; 145 MiB data, 778 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:20:33.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:33 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.0"}]: dispatch 2024-03-26T04:20:33.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:33 smithi183 ceph-mon[33745]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.0"}]: dispatch 2024-03-26T04:20:33.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:33 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.1"}]: dispatch 2024-03-26T04:20:33.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:33 smithi183 ceph-mon[33745]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.1"}]: dispatch 2024-03-26T04:20:33.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:33 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.2"}]: dispatch 2024-03-26T04:20:33.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:33 smithi183 ceph-mon[33745]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.2"}]: dispatch 2024-03-26T04:20:33.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:33 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.3"}]: dispatch 2024-03-26T04:20:33.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:33 smithi183 ceph-mon[33745]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.3"}]: dispatch 2024-03-26T04:20:33.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:33 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.4"}]: dispatch 2024-03-26T04:20:33.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:33 smithi183 ceph-mon[33745]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.4"}]: dispatch 2024-03-26T04:20:33.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:33 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.5"}]: dispatch 2024-03-26T04:20:33.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:33 smithi183 ceph-mon[33745]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.5"}]: dispatch 2024-03-26T04:20:33.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:33 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.6"}]: dispatch 2024-03-26T04:20:33.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:33 smithi183 ceph-mon[33745]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.6"}]: dispatch 2024-03-26T04:20:33.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:33 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.7"}]: dispatch 2024-03-26T04:20:33.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:33 smithi183 ceph-mon[33745]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.7"}]: dispatch 2024-03-26T04:20:33.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:33 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.8"}]: dispatch 2024-03-26T04:20:33.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:33 smithi183 ceph-mon[33745]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.8"}]: dispatch 2024-03-26T04:20:33.544 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:33 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.9"}]: dispatch 2024-03-26T04:20:33.544 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:33 smithi183 ceph-mon[33745]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "19.9"}]: dispatch 2024-03-26T04:20:33.544 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:33 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:33.544 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:33 smithi183 ceph-mon[33745]: from='client.40450 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsEC_smithi016-74578-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsEC_smithi016-74578-10"}]': finished 2024-03-26T04:20:33.544 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:33 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:33.544 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:33 smithi183 ceph-mon[33745]: osdmap e108: 8 total, 8 up, 8 in 2024-03-26T04:20:33.544 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:33 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1237666305' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi016-74285-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:33.544 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:33 smithi183 ceph-mon[33745]: from='client.43087 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi016-74285-6","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:33.544 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:33 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:20:33.544 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:33 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:34.168 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[29101]: 19.0 deep-scrub starts 2024-03-26T04:20:34.168 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[29101]: 19.0 deep-scrub ok 2024-03-26T04:20:34.168 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[29101]: 19.6 deep-scrub starts 2024-03-26T04:20:34.168 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[29101]: 19.6 deep-scrub ok 2024-03-26T04:20:34.168 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[29101]: 19.3 deep-scrub starts 2024-03-26T04:20:34.169 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[29101]: 19.3 deep-scrub ok 2024-03-26T04:20:34.169 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:20:34.169 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[29101]: 19.9 deep-scrub starts 2024-03-26T04:20:34.169 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[29101]: 19.9 deep-scrub ok 2024-03-26T04:20:34.169 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[29101]: 19.8 deep-scrub starts 2024-03-26T04:20:34.169 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[29101]: 19.8 deep-scrub ok 2024-03-26T04:20:34.169 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:20:34.169 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[29101]: 19.5 deep-scrub starts 2024-03-26T04:20:34.169 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[29101]: 19.5 deep-scrub ok 2024-03-26T04:20:34.169 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[29101]: pgmap v122: 524 pgs: 40 creating+peering, 1 active+clean+scrubbing+deep, 483 active+clean; 217 MiB data, 976 MiB used, 714 GiB / 715 GiB avail; 18 MiB/s wr, 0 op/s 2024-03-26T04:20:34.169 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[29101]: from='client.43087 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi016-74285-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:34.169 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-9", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:20:34.169 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[29101]: osdmap e109: 8 total, 8 up, 8 in 2024-03-26T04:20:34.169 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-9"}]: dispatch 2024-03-26T04:20:34.170 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/341636031' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi016-74264-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:34.170 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-9"}]: dispatch 2024-03-26T04:20:34.170 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[29101]: from='client.44575 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi016-74264-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:34.170 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:34.170 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[37215]: 19.0 deep-scrub starts 2024-03-26T04:20:34.170 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[37215]: 19.0 deep-scrub ok 2024-03-26T04:20:34.170 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[37215]: 19.6 deep-scrub starts 2024-03-26T04:20:34.170 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[37215]: 19.6 deep-scrub ok 2024-03-26T04:20:34.170 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[37215]: 19.3 deep-scrub starts 2024-03-26T04:20:34.170 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[37215]: 19.3 deep-scrub ok 2024-03-26T04:20:34.170 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:20:34.171 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[37215]: 19.9 deep-scrub starts 2024-03-26T04:20:34.171 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[37215]: 19.9 deep-scrub ok 2024-03-26T04:20:34.171 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[37215]: 19.8 deep-scrub starts 2024-03-26T04:20:34.171 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[37215]: 19.8 deep-scrub ok 2024-03-26T04:20:34.171 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:20:34.171 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[37215]: 19.5 deep-scrub starts 2024-03-26T04:20:34.171 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[37215]: 19.5 deep-scrub ok 2024-03-26T04:20:34.171 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[37215]: pgmap v122: 524 pgs: 40 creating+peering, 1 active+clean+scrubbing+deep, 483 active+clean; 217 MiB data, 976 MiB used, 714 GiB / 715 GiB avail; 18 MiB/s wr, 0 op/s 2024-03-26T04:20:34.171 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[37215]: from='client.43087 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi016-74285-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:34.171 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-9", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:20:34.171 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[37215]: osdmap e109: 8 total, 8 up, 8 in 2024-03-26T04:20:34.171 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-9"}]: dispatch 2024-03-26T04:20:34.171 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/341636031' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi016-74264-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:34.172 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-9"}]: dispatch 2024-03-26T04:20:34.172 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[37215]: from='client.44575 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi016-74264-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:34.172 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:34 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:34.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:34 smithi183 ceph-mon[33745]: 19.0 deep-scrub starts 2024-03-26T04:20:34.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:34 smithi183 ceph-mon[33745]: 19.0 deep-scrub ok 2024-03-26T04:20:34.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:34 smithi183 ceph-mon[33745]: 19.6 deep-scrub starts 2024-03-26T04:20:34.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:34 smithi183 ceph-mon[33745]: 19.6 deep-scrub ok 2024-03-26T04:20:34.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:34 smithi183 ceph-mon[33745]: 19.3 deep-scrub starts 2024-03-26T04:20:34.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:34 smithi183 ceph-mon[33745]: 19.3 deep-scrub ok 2024-03-26T04:20:34.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:34 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:20:34.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:34 smithi183 ceph-mon[33745]: 19.9 deep-scrub starts 2024-03-26T04:20:34.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:34 smithi183 ceph-mon[33745]: 19.9 deep-scrub ok 2024-03-26T04:20:34.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:34 smithi183 ceph-mon[33745]: 19.8 deep-scrub starts 2024-03-26T04:20:34.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:34 smithi183 ceph-mon[33745]: 19.8 deep-scrub ok 2024-03-26T04:20:34.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:34 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-9", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:20:34.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:34 smithi183 ceph-mon[33745]: 19.5 deep-scrub starts 2024-03-26T04:20:34.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:34 smithi183 ceph-mon[33745]: 19.5 deep-scrub ok 2024-03-26T04:20:34.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:34 smithi183 ceph-mon[33745]: pgmap v122: 524 pgs: 40 creating+peering, 1 active+clean+scrubbing+deep, 483 active+clean; 217 MiB data, 976 MiB used, 714 GiB / 715 GiB avail; 18 MiB/s wr, 0 op/s 2024-03-26T04:20:34.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:34 smithi183 ceph-mon[33745]: from='client.43087 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafePP_smithi016-74285-6","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:34.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:34 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-9", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:20:34.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:34 smithi183 ceph-mon[33745]: osdmap e109: 8 total, 8 up, 8 in 2024-03-26T04:20:34.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:34 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-9"}]: dispatch 2024-03-26T04:20:34.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:34 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/341636031' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi016-74264-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:34.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:34 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-9"}]: dispatch 2024-03-26T04:20:34.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:34 smithi183 ceph-mon[33745]: from='client.44575 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattr_smithi016-74264-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:34.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:34 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:35.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:35 smithi183 ceph-mon[33745]: 19.2 deep-scrub starts 2024-03-26T04:20:35.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:35 smithi183 ceph-mon[33745]: 19.2 deep-scrub ok 2024-03-26T04:20:35.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:35 smithi183 ceph-mon[33745]: 19.4 deep-scrub starts 2024-03-26T04:20:35.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:35 smithi183 ceph-mon[33745]: 19.4 deep-scrub ok 2024-03-26T04:20:35.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:35 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-9"}]': finished 2024-03-26T04:20:35.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:35 smithi183 ceph-mon[33745]: from='client.44575 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi016-74264-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:35.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:35 smithi183 ceph-mon[33745]: osdmap e110: 8 total, 8 up, 8 in 2024-03-26T04:20:35.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:35 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-9", "mode": "writeback"}]: dispatch 2024-03-26T04:20:35.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:35 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-9", "mode": "writeback"}]: dispatch 2024-03-26T04:20:35.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:35 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1669091670' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:35.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:35 smithi183 ceph-mon[33745]: from='client.45790 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:35.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:35 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:35.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:35 smithi016 ceph-mon[37215]: 19.2 deep-scrub starts 2024-03-26T04:20:35.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:35 smithi016 ceph-mon[37215]: 19.2 deep-scrub ok 2024-03-26T04:20:35.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:35 smithi016 ceph-mon[37215]: 19.4 deep-scrub starts 2024-03-26T04:20:35.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:35 smithi016 ceph-mon[37215]: 19.4 deep-scrub ok 2024-03-26T04:20:35.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:35 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-9"}]': finished 2024-03-26T04:20:35.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:35 smithi016 ceph-mon[37215]: from='client.44575 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi016-74264-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:35.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:35 smithi016 ceph-mon[37215]: osdmap e110: 8 total, 8 up, 8 in 2024-03-26T04:20:35.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:35 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-9", "mode": "writeback"}]: dispatch 2024-03-26T04:20:35.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:35 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-9", "mode": "writeback"}]: dispatch 2024-03-26T04:20:35.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:35 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1669091670' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:35.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:35 smithi016 ceph-mon[37215]: from='client.45790 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:35.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:35 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:35.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:35 smithi016 ceph-mon[29101]: 19.2 deep-scrub starts 2024-03-26T04:20:35.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:35 smithi016 ceph-mon[29101]: 19.2 deep-scrub ok 2024-03-26T04:20:35.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:35 smithi016 ceph-mon[29101]: 19.4 deep-scrub starts 2024-03-26T04:20:35.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:35 smithi016 ceph-mon[29101]: 19.4 deep-scrub ok 2024-03-26T04:20:35.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:35 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-9"}]': finished 2024-03-26T04:20:35.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:35 smithi016 ceph-mon[29101]: from='client.44575 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattr_smithi016-74264-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:35.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:35 smithi016 ceph-mon[29101]: osdmap e110: 8 total, 8 up, 8 in 2024-03-26T04:20:35.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:35 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-9", "mode": "writeback"}]: dispatch 2024-03-26T04:20:35.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:35 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-9", "mode": "writeback"}]: dispatch 2024-03-26T04:20:35.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:35 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1669091670' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:35.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:35 smithi016 ceph-mon[29101]: from='client.45790 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:35.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:35 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:36.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:36 smithi016 ceph-mon[29101]: 19.1 deep-scrub starts 2024-03-26T04:20:36.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:36 smithi016 ceph-mon[29101]: 19.1 deep-scrub ok 2024-03-26T04:20:36.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:36 smithi016 ceph-mon[29101]: pgmap v125: 588 pgs: 128 unknown, 8 creating+peering, 1 active+clean+scrubbing+deep, 451 active+clean; 217 MiB data, 976 MiB used, 714 GiB / 715 GiB avail; 980 B/s rd, 17 MiB/s wr, 6 op/s 2024-03-26T04:20:36.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:36 smithi016 ceph-mon[29101]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:20:36.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:36 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-9", "mode": "writeback"}]': finished 2024-03-26T04:20:36.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:36 smithi016 ceph-mon[29101]: from='client.45790 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:36.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:36 smithi016 ceph-mon[29101]: osdmap e111: 8 total, 8 up, 8 in 2024-03-26T04:20:36.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:36 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3539257470' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi016-74285-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:36.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:36 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:36.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:36 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:20:36.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:36 smithi016 ceph-mon[29101]: from='client.46873 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi016-74285-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:36.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:36 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:20:36.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:36 smithi016 ceph-mon[29101]: Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:20:36.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:36 smithi016 ceph-mon[29101]: from='client.46873 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi016-74285-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:36.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:36 smithi016 ceph-mon[37215]: 19.1 deep-scrub starts 2024-03-26T04:20:36.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:36 smithi016 ceph-mon[37215]: 19.1 deep-scrub ok 2024-03-26T04:20:36.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:36 smithi016 ceph-mon[37215]: pgmap v125: 588 pgs: 128 unknown, 8 creating+peering, 1 active+clean+scrubbing+deep, 451 active+clean; 217 MiB data, 976 MiB used, 714 GiB / 715 GiB avail; 980 B/s rd, 17 MiB/s wr, 6 op/s 2024-03-26T04:20:36.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:36 smithi016 ceph-mon[37215]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:20:36.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:36 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-9", "mode": "writeback"}]': finished 2024-03-26T04:20:36.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:36 smithi016 ceph-mon[37215]: from='client.45790 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:36.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:36 smithi016 ceph-mon[37215]: osdmap e111: 8 total, 8 up, 8 in 2024-03-26T04:20:36.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:36 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3539257470' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi016-74285-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:36.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:36 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:36.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:36 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:20:36.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:36 smithi016 ceph-mon[37215]: from='client.46873 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi016-74285-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:36.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:36 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:20:36.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:36 smithi016 ceph-mon[37215]: Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:20:36.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:36 smithi016 ceph-mon[37215]: from='client.46873 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi016-74285-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:37.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:36 smithi183 ceph-mon[33745]: 19.1 deep-scrub starts 2024-03-26T04:20:37.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:36 smithi183 ceph-mon[33745]: 19.1 deep-scrub ok 2024-03-26T04:20:37.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:36 smithi183 ceph-mon[33745]: pgmap v125: 588 pgs: 128 unknown, 8 creating+peering, 1 active+clean+scrubbing+deep, 451 active+clean; 217 MiB data, 976 MiB used, 714 GiB / 715 GiB avail; 980 B/s rd, 17 MiB/s wr, 6 op/s 2024-03-26T04:20:37.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:36 smithi183 ceph-mon[33745]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:20:37.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:36 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-9", "mode": "writeback"}]': finished 2024-03-26T04:20:37.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:36 smithi183 ceph-mon[33745]: from='client.45790 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74595-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:37.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:36 smithi183 ceph-mon[33745]: osdmap e111: 8 total, 8 up, 8 in 2024-03-26T04:20:37.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:36 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3539257470' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi016-74285-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:37.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:36 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:37.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:36 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:20:37.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:36 smithi183 ceph-mon[33745]: from='client.46873 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi016-74285-7","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:37.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:36 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:20:37.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:36 smithi183 ceph-mon[33745]: Health check update: 8 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:20:37.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:36 smithi183 ceph-mon[33745]: from='client.46873 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP_smithi016-74285-7","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:37.089 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-03-26T04:20:37.089 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: [----------] Global test environment set-up. 2024-03-26T04:20:37.089 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-03-26T04:20:37.089 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-03-26T04:20:37.090 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-03-26T04:20:37.090 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (1 ms total) 2024-03-26T04:20:37.090 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: 2024-03-26T04:20:37.090 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-03-26T04:20:37.090 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-03-26T04:20:37.090 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-03-26T04:20:37.090 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: 2024-03-26T04:19:48.683+0000 7f5679239780 -1 failed for service _ceph-mon._tcp 2024-03-26T04:20:37.090 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: 2024-03-26T04:19:48.683+0000 7f5679239780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-03-26T04:20:37.090 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (65 ms) 2024-03-26T04:20:37.090 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-03-26T04:20:37.090 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2013 ms) 2024-03-26T04:20:37.090 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2078 ms total) 2024-03-26T04:20:37.091 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: 2024-03-26T04:20:37.091 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-03-26T04:20:37.091 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-03-26T04:20:37.091 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: started 0x55ee5ff09600 2024-03-26T04:20:37.091 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: started 0x55ee602e1520 2024-03-26T04:20:37.091 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: started 2 aios 2024-03-26T04:20:37.091 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: waiting 0x55ee5ff09600 2024-03-26T04:20:37.091 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: waiting 0x55ee602e1520 2024-03-26T04:20:37.091 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: done. 2024-03-26T04:20:37.091 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (12105 ms) 2024-03-26T04:20:37.091 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (12105 ms total) 2024-03-26T04:20:37.091 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: 2024-03-26T04:20:37.091 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-03-26T04:20:37.092 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-03-26T04:20:37.092 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-03-26T04:20:37.092 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-03-26T04:20:37.092 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: [ OK ] LibRadosMisc.Exec (101 ms) 2024-03-26T04:20:37.092 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-03-26T04:20:37.092 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (4 ms) 2024-03-26T04:20:37.092 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-03-26T04:20:37.092 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (1016 ms) 2024-03-26T04:20:37.092 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-03-26T04:20:37.092 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: [ OK ] LibRadosMisc.Applications (6853 ms) 2024-03-26T04:20:37.093 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-03-26T04:20:37.093 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-03-26T04:20:37.093 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-03-26T04:20:37.093 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-03-26T04:20:37.093 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-03-26T04:20:37.093 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (22916 ms) 2024-03-26T04:20:37.093 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: [----------] 8 tests from LibRadosMisc (30891 ms total) 2024-03-26T04:20:37.093 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: 2024-03-26T04:20:37.093 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: [----------] Global test environment tear-down 2024-03-26T04:20:37.093 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (48446 ms total) 2024-03-26T04:20:37.093 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc: [ PASSED ] 12 tests. 2024-03-26T04:20:37.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:37 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:20:37.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:37 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1669091670' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74595-10", "tierpool":"test-rados-api-smithi016-74595-10-cache", "force_nonempty":""}]: dispatch 2024-03-26T04:20:37.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:37 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3001077418' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi016-74264-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:37.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:37 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-9"}]: dispatch 2024-03-26T04:20:37.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:37 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:37.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:37 smithi016 ceph-mon[29101]: osdmap e112: 8 total, 8 up, 8 in 2024-03-26T04:20:37.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:37 smithi016 ceph-mon[29101]: from='client.45790 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74595-10", "tierpool":"test-rados-api-smithi016-74595-10-cache", "force_nonempty":""}]: dispatch 2024-03-26T04:20:37.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:37 smithi016 ceph-mon[29101]: from='client.48148 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi016-74264-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:37.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:37 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-9"}]: dispatch 2024-03-26T04:20:37.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:37 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:20:37.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:37 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:20:37.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:37 smithi016 ceph-mon[29101]: 19.7 deep-scrub starts 2024-03-26T04:20:37.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:37 smithi016 ceph-mon[29101]: 19.7 deep-scrub ok 2024-03-26T04:20:37.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:37 smithi016 ceph-mon[29101]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:20:37.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:37 smithi016 ceph-mon[29101]: from='client.45790 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74595-10", "tierpool":"test-rados-api-smithi016-74595-10-cache", "force_nonempty":""}]': finished 2024-03-26T04:20:37.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:37 smithi016 ceph-mon[29101]: from='client.48148 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi016-74264-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:37.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:37 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-9"}]': finished 2024-03-26T04:20:37.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:37 smithi016 ceph-mon[29101]: osdmap e113: 8 total, 8 up, 8 in 2024-03-26T04:20:37.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:37 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1669091670' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74595-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:37.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:37 smithi016 ceph-mon[29101]: from='client.45790 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74595-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:37.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:37 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:37.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:37 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:20:37.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:37 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1669091670' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74595-10", "tierpool":"test-rados-api-smithi016-74595-10-cache", "force_nonempty":""}]: dispatch 2024-03-26T04:20:37.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:37 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3001077418' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi016-74264-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:37.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:37 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-9"}]: dispatch 2024-03-26T04:20:37.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:37 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:37.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:37 smithi016 ceph-mon[37215]: osdmap e112: 8 total, 8 up, 8 in 2024-03-26T04:20:37.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:37 smithi016 ceph-mon[37215]: from='client.45790 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74595-10", "tierpool":"test-rados-api-smithi016-74595-10-cache", "force_nonempty":""}]: dispatch 2024-03-26T04:20:37.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:37 smithi016 ceph-mon[37215]: from='client.48148 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi016-74264-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:37.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:37 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-9"}]: dispatch 2024-03-26T04:20:37.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:37 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:20:37.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:37 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:20:37.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:37 smithi016 ceph-mon[37215]: 19.7 deep-scrub starts 2024-03-26T04:20:37.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:37 smithi016 ceph-mon[37215]: 19.7 deep-scrub ok 2024-03-26T04:20:37.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:37 smithi016 ceph-mon[37215]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:20:37.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:37 smithi016 ceph-mon[37215]: from='client.45790 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74595-10", "tierpool":"test-rados-api-smithi016-74595-10-cache", "force_nonempty":""}]': finished 2024-03-26T04:20:37.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:37 smithi016 ceph-mon[37215]: from='client.48148 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi016-74264-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:37.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:37 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-9"}]': finished 2024-03-26T04:20:37.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:37 smithi016 ceph-mon[37215]: osdmap e113: 8 total, 8 up, 8 in 2024-03-26T04:20:37.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:37 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1669091670' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74595-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:37.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:37 smithi016 ceph-mon[37215]: from='client.45790 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74595-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:37.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:37 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:38.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:37 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:20:38.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:37 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1669091670' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74595-10", "tierpool":"test-rados-api-smithi016-74595-10-cache", "force_nonempty":""}]: dispatch 2024-03-26T04:20:38.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:37 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3001077418' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi016-74264-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:38.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:37 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-9"}]: dispatch 2024-03-26T04:20:38.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:37 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:38.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:37 smithi183 ceph-mon[33745]: osdmap e112: 8 total, 8 up, 8 in 2024-03-26T04:20:38.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:37 smithi183 ceph-mon[33745]: from='client.45790 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74595-10", "tierpool":"test-rados-api-smithi016-74595-10-cache", "force_nonempty":""}]: dispatch 2024-03-26T04:20:38.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:37 smithi183 ceph-mon[33745]: from='client.48148 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrIter_smithi016-74264-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:38.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:37 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-9"}]: dispatch 2024-03-26T04:20:38.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:37 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:20:38.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:37 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:20:38.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:37 smithi183 ceph-mon[33745]: 19.7 deep-scrub starts 2024-03-26T04:20:38.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:37 smithi183 ceph-mon[33745]: 19.7 deep-scrub ok 2024-03-26T04:20:38.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:37 smithi183 ceph-mon[33745]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:20:38.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:37 smithi183 ceph-mon[33745]: from='client.45790 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74595-10", "tierpool":"test-rados-api-smithi016-74595-10-cache", "force_nonempty":""}]': finished 2024-03-26T04:20:38.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:37 smithi183 ceph-mon[33745]: from='client.48148 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrIter_smithi016-74264-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:38.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:37 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-9"}]': finished 2024-03-26T04:20:38.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:37 smithi183 ceph-mon[33745]: osdmap e113: 8 total, 8 up, 8 in 2024-03-26T04:20:38.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:37 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1669091670' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74595-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:38.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:37 smithi183 ceph-mon[33745]: from='client.45790 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74595-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:38.044 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:37 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:38.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:20:38 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:20:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:20:38.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:38 smithi016 ceph-mon[29101]: pgmap v128: 588 pgs: 145 unknown, 19 creating+peering, 1 active+clean+scrubbing+deep, 423 active+clean; 217 MiB data, 977 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:20:38.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:38 smithi016 ceph-mon[29101]: from='client.45790 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74595-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:38.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:38 smithi016 ceph-mon[29101]: osdmap e114: 8 total, 8 up, 8 in 2024-03-26T04:20:38.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:38 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1669091670' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74595-10", "tierpool":"test-rados-api-smithi016-74595-10-cache"}]: dispatch 2024-03-26T04:20:38.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:38 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2330325028' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi016-74285-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:38.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:38 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:38.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:38 smithi016 ceph-mon[29101]: from='client.45790 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74595-10", "tierpool":"test-rados-api-smithi016-74595-10-cache"}]: dispatch 2024-03-26T04:20:38.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:38 smithi016 ceph-mon[29101]: from='client.48589 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi016-74285-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:38.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:38 smithi016 ceph-mon[37215]: pgmap v128: 588 pgs: 145 unknown, 19 creating+peering, 1 active+clean+scrubbing+deep, 423 active+clean; 217 MiB data, 977 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:20:38.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:38 smithi016 ceph-mon[37215]: from='client.45790 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74595-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:38.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:38 smithi016 ceph-mon[37215]: osdmap e114: 8 total, 8 up, 8 in 2024-03-26T04:20:38.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:38 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1669091670' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74595-10", "tierpool":"test-rados-api-smithi016-74595-10-cache"}]: dispatch 2024-03-26T04:20:38.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:38 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2330325028' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi016-74285-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:38.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:38 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:38.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:38 smithi016 ceph-mon[37215]: from='client.45790 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74595-10", "tierpool":"test-rados-api-smithi016-74595-10-cache"}]: dispatch 2024-03-26T04:20:38.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:38 smithi016 ceph-mon[37215]: from='client.48589 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi016-74285-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:39.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:38 smithi183 ceph-mon[33745]: pgmap v128: 588 pgs: 145 unknown, 19 creating+peering, 1 active+clean+scrubbing+deep, 423 active+clean; 217 MiB data, 977 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:20:39.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:38 smithi183 ceph-mon[33745]: from='client.45790 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74595-10-cache", "mode":"readonly", "yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:39.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:38 smithi183 ceph-mon[33745]: osdmap e114: 8 total, 8 up, 8 in 2024-03-26T04:20:39.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:38 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1669091670' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74595-10", "tierpool":"test-rados-api-smithi016-74595-10-cache"}]: dispatch 2024-03-26T04:20:39.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:38 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2330325028' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi016-74285-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:39.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:38 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:39.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:38 smithi183 ceph-mon[33745]: from='client.45790 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74595-10", "tierpool":"test-rados-api-smithi016-74595-10-cache"}]: dispatch 2024-03-26T04:20:39.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:38 smithi183 ceph-mon[33745]: from='client.48589 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi016-74285-8","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:40.079 INFO:tasks.workunit.client.0.smithi016.stdout: misc: Running main() from gmock_main.cc 2024-03-26T04:20:40.079 INFO:tasks.workunit.client.0.smithi016.stdout: misc: [==========] Running 12 tests from 1 test suite. 2024-03-26T04:20:40.080 INFO:tasks.workunit.client.0.smithi016.stdout: misc: [----------] Global test environment set-up. 2024-03-26T04:20:40.080 INFO:tasks.workunit.client.0.smithi016.stdout: misc: [----------] 12 tests from NeoRadosMisc 2024-03-26T04:20:40.080 INFO:tasks.workunit.client.0.smithi016.stdout: misc: [ RUN ] NeoRadosMisc.Version 2024-03-26T04:20:40.080 INFO:tasks.workunit.client.0.smithi016.stdout: misc: [ OK ] NeoRadosMisc.Version (9282 ms) 2024-03-26T04:20:40.080 INFO:tasks.workunit.client.0.smithi016.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2024-03-26T04:20:40.080 INFO:tasks.workunit.client.0.smithi016.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (3495 ms) 2024-03-26T04:20:40.080 INFO:tasks.workunit.client.0.smithi016.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2024-03-26T04:20:40.080 INFO:tasks.workunit.client.0.smithi016.stdout: misc: [ OK ] NeoRadosMisc.LongName (5248 ms) 2024-03-26T04:20:40.080 INFO:tasks.workunit.client.0.smithi016.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2024-03-26T04:20:40.080 INFO:tasks.workunit.client.0.smithi016.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (5926 ms) 2024-03-26T04:20:40.080 INFO:tasks.workunit.client.0.smithi016.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2024-03-26T04:20:40.080 INFO:tasks.workunit.client.0.smithi016.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (3626 ms) 2024-03-26T04:20:40.080 INFO:tasks.workunit.client.0.smithi016.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2024-03-26T04:20:40.081 INFO:tasks.workunit.client.0.smithi016.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (3290 ms) 2024-03-26T04:20:40.081 INFO:tasks.workunit.client.0.smithi016.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2024-03-26T04:20:40.081 INFO:tasks.workunit.client.0.smithi016.stdout: misc: [ OK ] NeoRadosMisc.Exec (4384 ms) 2024-03-26T04:20:40.081 INFO:tasks.workunit.client.0.smithi016.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2024-03-26T04:20:40.081 INFO:tasks.workunit.client.0.smithi016.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (3644 ms) 2024-03-26T04:20:40.081 INFO:tasks.workunit.client.0.smithi016.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2024-03-26T04:20:40.081 INFO:tasks.workunit.client.0.smithi016.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (3997 ms) 2024-03-26T04:20:40.081 INFO:tasks.workunit.client.0.smithi016.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2024-03-26T04:20:40.081 INFO:tasks.workunit.client.0.smithi016.stdout: misc: [ OK ] NeoRadosMisc.BigObject (3298 ms) 2024-03-26T04:20:40.081 INFO:tasks.workunit.client.0.smithi016.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2024-03-26T04:20:40.081 INFO:tasks.workunit.client.0.smithi016.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (1923 ms) 2024-03-26T04:20:40.081 INFO:tasks.workunit.client.0.smithi016.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2024-03-26T04:20:40.081 INFO:tasks.workunit.client.0.smithi016.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (3011 ms) 2024-03-26T04:20:40.081 INFO:tasks.workunit.client.0.smithi016.stdout: misc: [----------] 12 tests from NeoRadosMisc (51124 ms total) 2024-03-26T04:20:40.082 INFO:tasks.workunit.client.0.smithi016.stdout: misc: 2024-03-26T04:20:40.082 INFO:tasks.workunit.client.0.smithi016.stdout: misc: [----------] Global test environment tear-down 2024-03-26T04:20:40.082 INFO:tasks.workunit.client.0.smithi016.stdout: misc: [==========] 12 tests from 1 test suite ran. (51124 ms total) 2024-03-26T04:20:40.082 INFO:tasks.workunit.client.0.smithi016.stdout: misc: [ PASSED ] 12 tests. 2024-03-26T04:20:40.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:40 smithi016 ceph-mon[29101]: pgmap v131: 556 pgs: 101 unknown, 51 creating+peering, 404 active+clean; 217 MiB data, 984 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-03-26T04:20:40.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:40 smithi016 ceph-mon[29101]: from='client.45790 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74595-10", "tierpool":"test-rados-api-smithi016-74595-10-cache"}]': finished 2024-03-26T04:20:40.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:40 smithi016 ceph-mon[29101]: from='client.48589 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi016-74285-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:40.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:40 smithi016 ceph-mon[29101]: osdmap e115: 8 total, 8 up, 8 in 2024-03-26T04:20:40.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:40 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:40.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:40 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2394375358' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi016-74264-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:40.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:40 smithi016 ceph-mon[29101]: from='client.56075 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi016-74264-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:40.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:40 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:40.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:40 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:40.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:40 smithi016 ceph-mon[37215]: pgmap v131: 556 pgs: 101 unknown, 51 creating+peering, 404 active+clean; 217 MiB data, 984 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-03-26T04:20:40.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:40 smithi016 ceph-mon[37215]: from='client.45790 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74595-10", "tierpool":"test-rados-api-smithi016-74595-10-cache"}]': finished 2024-03-26T04:20:40.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:40 smithi016 ceph-mon[37215]: from='client.48589 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi016-74285-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:40.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:40 smithi016 ceph-mon[37215]: osdmap e115: 8 total, 8 up, 8 in 2024-03-26T04:20:40.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:40 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:40.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:40 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2394375358' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi016-74264-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:40.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:40 smithi016 ceph-mon[37215]: from='client.56075 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi016-74264-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:40.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:40 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:40.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:40 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:40.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:40 smithi183 ceph-mon[33745]: pgmap v131: 556 pgs: 101 unknown, 51 creating+peering, 404 active+clean; 217 MiB data, 984 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-03-26T04:20:40.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:40 smithi183 ceph-mon[33745]: from='client.45790 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74595-10", "tierpool":"test-rados-api-smithi016-74595-10-cache"}]': finished 2024-03-26T04:20:40.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:40 smithi183 ceph-mon[33745]: from='client.48589 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP2_smithi016-74285-8","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:40.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:40 smithi183 ceph-mon[33745]: osdmap e115: 8 total, 8 up, 8 in 2024-03-26T04:20:40.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:40 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:40.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:40 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2394375358' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi016-74264-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:40.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:40 smithi183 ceph-mon[33745]: from='client.56075 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsComplete_smithi016-74264-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:40.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:40 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:40.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:40 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:41.074 INFO:tasks.workunit.client.0.smithi016.stdout: api_pool: Running main() from gmock_main.cc 2024-03-26T04:20:41.074 INFO:tasks.workunit.client.0.smithi016.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-03-26T04:20:41.074 INFO:tasks.workunit.client.0.smithi016.stdout: api_pool: [----------] Global test environment set-up. 2024-03-26T04:20:41.074 INFO:tasks.workunit.client.0.smithi016.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-03-26T04:20:41.074 INFO:tasks.workunit.client.0.smithi016.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-03-26T04:20:41.075 INFO:tasks.workunit.client.0.smithi016.stdout: api_pool: [ OK ] LibRadosPools.PoolList (11125 ms) 2024-03-26T04:20:41.075 INFO:tasks.workunit.client.0.smithi016.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-03-26T04:20:41.075 INFO:tasks.workunit.client.0.smithi016.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (3040 ms) 2024-03-26T04:20:41.075 INFO:tasks.workunit.client.0.smithi016.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-03-26T04:20:41.075 INFO:tasks.workunit.client.0.smithi016.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (6497 ms) 2024-03-26T04:20:41.075 INFO:tasks.workunit.client.0.smithi016.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-03-26T04:20:41.075 INFO:tasks.workunit.client.0.smithi016.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (3613 ms) 2024-03-26T04:20:41.075 INFO:tasks.workunit.client.0.smithi016.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-03-26T04:20:41.075 INFO:tasks.workunit.client.0.smithi016.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (5014 ms) 2024-03-26T04:20:41.075 INFO:tasks.workunit.client.0.smithi016.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-03-26T04:20:41.076 INFO:tasks.workunit.client.0.smithi016.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (5290 ms) 2024-03-26T04:20:41.076 INFO:tasks.workunit.client.0.smithi016.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-03-26T04:20:41.076 INFO:tasks.workunit.client.0.smithi016.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (4641 ms) 2024-03-26T04:20:41.076 INFO:tasks.workunit.client.0.smithi016.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-03-26T04:20:41.076 INFO:tasks.workunit.client.0.smithi016.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (5259 ms) 2024-03-26T04:20:41.076 INFO:tasks.workunit.client.0.smithi016.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-03-26T04:20:41.076 INFO:tasks.workunit.client.0.smithi016.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (7957 ms) 2024-03-26T04:20:41.076 INFO:tasks.workunit.client.0.smithi016.stdout: api_pool: [----------] 9 tests from LibRadosPools (52436 ms total) 2024-03-26T04:20:41.076 INFO:tasks.workunit.client.0.smithi016.stdout: api_pool: 2024-03-26T04:20:41.076 INFO:tasks.workunit.client.0.smithi016.stdout: api_pool: [----------] Global test environment tear-down 2024-03-26T04:20:41.076 INFO:tasks.workunit.client.0.smithi016.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (52436 ms total) 2024-03-26T04:20:41.076 INFO:tasks.workunit.client.0.smithi016.stdout: api_pool: [ PASSED ] 9 tests. 2024-03-26T04:20:41.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:41 smithi016 ceph-mon[29101]: from='client.56075 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi016-74264-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:41.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:41 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:41.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:41 smithi016 ceph-mon[29101]: osdmap e116: 8 total, 8 up, 8 in 2024-03-26T04:20:41.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:41 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:41.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:41 smithi016 ceph-mon[29101]: osdmap e117: 8 total, 8 up, 8 in 2024-03-26T04:20:41.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:41 smithi016 ceph-mon[37215]: from='client.56075 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi016-74264-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:41.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:41 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:41.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:41 smithi016 ceph-mon[37215]: osdmap e116: 8 total, 8 up, 8 in 2024-03-26T04:20:41.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:41 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:41.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:41 smithi016 ceph-mon[37215]: osdmap e117: 8 total, 8 up, 8 in 2024-03-26T04:20:41.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:41 smithi183 ceph-mon[33745]: from='client.56075 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsComplete_smithi016-74264-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:41.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:41 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:41.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:41 smithi183 ceph-mon[33745]: osdmap e116: 8 total, 8 up, 8 in 2024-03-26T04:20:41.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:41 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:41.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:41 smithi183 ceph-mon[33745]: osdmap e117: 8 total, 8 up, 8 in 2024-03-26T04:20:42.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:42 smithi016 ceph-mon[29101]: pgmap v134: 524 pgs: 114 unknown, 14 creating+peering, 396 active+clean; 217 MiB data, 984 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-03-26T04:20:42.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:42 smithi016 ceph-mon[29101]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:20:42.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:42 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3707580190' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi016-74285-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:42.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:42 smithi016 ceph-mon[29101]: from='client.48604 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi016-74285-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:42.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:42 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:20:42.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:42 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:42.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:42 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:20:42.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:42 smithi016 ceph-mon[29101]: from='client.48604 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi016-74285-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:42.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:42 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-11", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:20:42.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:42 smithi016 ceph-mon[29101]: osdmap e118: 8 total, 8 up, 8 in 2024-03-26T04:20:42.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:42 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-11"}]: dispatch 2024-03-26T04:20:42.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:42 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2934028528' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi016-74578-10"}]: dispatch 2024-03-26T04:20:42.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:42 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-11"}]: dispatch 2024-03-26T04:20:42.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:42 smithi016 ceph-mon[29101]: from='client.40450 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi016-74578-10"}]: dispatch 2024-03-26T04:20:42.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:42 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/273633222' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi016-74264-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:42.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:42 smithi016 ceph-mon[37215]: pgmap v134: 524 pgs: 114 unknown, 14 creating+peering, 396 active+clean; 217 MiB data, 984 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-03-26T04:20:42.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:42 smithi016 ceph-mon[37215]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:20:42.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:42 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3707580190' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi016-74285-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:42.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:42 smithi016 ceph-mon[37215]: from='client.48604 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi016-74285-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:42.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:42 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:20:42.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:42 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:42.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:42 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:20:42.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:42 smithi016 ceph-mon[37215]: from='client.48604 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi016-74285-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:42.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:42 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-11", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:20:42.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:42 smithi016 ceph-mon[37215]: osdmap e118: 8 total, 8 up, 8 in 2024-03-26T04:20:42.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:42 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-11"}]: dispatch 2024-03-26T04:20:42.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:42 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2934028528' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi016-74578-10"}]: dispatch 2024-03-26T04:20:42.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:42 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-11"}]: dispatch 2024-03-26T04:20:42.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:42 smithi016 ceph-mon[37215]: from='client.40450 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi016-74578-10"}]: dispatch 2024-03-26T04:20:42.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:42 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/273633222' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi016-74264-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:42.542 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:20:42 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:20:42.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:42 smithi183 ceph-mon[33745]: pgmap v134: 524 pgs: 114 unknown, 14 creating+peering, 396 active+clean; 217 MiB data, 984 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 1 op/s 2024-03-26T04:20:42.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:42 smithi183 ceph-mon[33745]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:20:42.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:42 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3707580190' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi016-74285-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:42.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:42 smithi183 ceph-mon[33745]: from='client.48604 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi016-74285-9","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:42.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:42 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:20:42.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:42 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:42.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:42 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-11", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:20:42.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:42 smithi183 ceph-mon[33745]: from='client.48604 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi016-74285-9","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:42.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:42 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-11", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:20:42.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:42 smithi183 ceph-mon[33745]: osdmap e118: 8 total, 8 up, 8 in 2024-03-26T04:20:42.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:42 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-11"}]: dispatch 2024-03-26T04:20:42.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:42 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2934028528' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi016-74578-10"}]: dispatch 2024-03-26T04:20:42.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:42 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-11"}]: dispatch 2024-03-26T04:20:42.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:42 smithi183 ceph-mon[33745]: from='client.40450 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi016-74578-10"}]: dispatch 2024-03-26T04:20:42.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:42 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/273633222' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafe_smithi016-74264-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:43.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:43 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:20:43.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:43 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:43.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:43 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-11"}]': finished 2024-03-26T04:20:43.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:43 smithi016 ceph-mon[29101]: from='client.40450 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi016-74578-10"}]': finished 2024-03-26T04:20:43.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:43 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/273633222' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi016-74264-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:43.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:43 smithi016 ceph-mon[29101]: osdmap e119: 8 total, 8 up, 8 in 2024-03-26T04:20:43.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:43 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-11", "mode": "writeback"}]: dispatch 2024-03-26T04:20:43.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:43 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2934028528' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi016-74578-10"}]: dispatch 2024-03-26T04:20:43.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:43 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-11", "mode": "writeback"}]: dispatch 2024-03-26T04:20:43.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:43 smithi016 ceph-mon[29101]: from='client.40450 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi016-74578-10"}]: dispatch 2024-03-26T04:20:43.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:43 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:20:43.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:43 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:43.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:43 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-11"}]': finished 2024-03-26T04:20:43.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:43 smithi016 ceph-mon[37215]: from='client.40450 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi016-74578-10"}]': finished 2024-03-26T04:20:43.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:43 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/273633222' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi016-74264-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:43.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:43 smithi016 ceph-mon[37215]: osdmap e119: 8 total, 8 up, 8 in 2024-03-26T04:20:43.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:43 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-11", "mode": "writeback"}]: dispatch 2024-03-26T04:20:43.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:43 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2934028528' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi016-74578-10"}]: dispatch 2024-03-26T04:20:43.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:43 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-11", "mode": "writeback"}]: dispatch 2024-03-26T04:20:43.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:43 smithi016 ceph-mon[37215]: from='client.40450 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi016-74578-10"}]: dispatch 2024-03-26T04:20:43.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:43 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:20:43.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:43 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:43.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:43 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-11"}]': finished 2024-03-26T04:20:43.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:43 smithi183 ceph-mon[33745]: from='client.40450 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsEC_smithi016-74578-10"}]': finished 2024-03-26T04:20:43.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:43 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/273633222' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafe_smithi016-74264-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:43.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:43 smithi183 ceph-mon[33745]: osdmap e119: 8 total, 8 up, 8 in 2024-03-26T04:20:43.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:43 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-11", "mode": "writeback"}]: dispatch 2024-03-26T04:20:43.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:43 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2934028528' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi016-74578-10"}]: dispatch 2024-03-26T04:20:43.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:43 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-11", "mode": "writeback"}]: dispatch 2024-03-26T04:20:43.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:43 smithi183 ceph-mon[33745]: from='client.40450 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi016-74578-10"}]: dispatch 2024-03-26T04:20:44.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:44 smithi016 ceph-mon[29101]: pgmap v137: 484 pgs: 38 creating+peering, 47 unknown, 399 active+clean; 217 MiB data, 1021 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-26T04:20:44.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:44 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:44.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:44 smithi016 ceph-mon[29101]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:20:44.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:44 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-11", "mode": "writeback"}]': finished 2024-03-26T04:20:44.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:44 smithi016 ceph-mon[29101]: from='client.40450 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi016-74578-10"}]': finished 2024-03-26T04:20:44.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:44 smithi016 ceph-mon[29101]: osdmap e120: 8 total, 8 up, 8 in 2024-03-26T04:20:44.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:44 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2129593090' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi016-74285-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:44.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:44 smithi016 ceph-mon[29101]: from='client.56102 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi016-74285-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:44.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:44 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/334241781' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi016-74578-15"}]: dispatch 2024-03-26T04:20:44.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:44 smithi016 ceph-mon[29101]: from='client.48637 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi016-74578-15"}]: dispatch 2024-03-26T04:20:44.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:44 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/334241781' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi016-74578-15"}]: dispatch 2024-03-26T04:20:44.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:44 smithi016 ceph-mon[29101]: from='client.48637 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi016-74578-15"}]: dispatch 2024-03-26T04:20:44.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:44 smithi016 ceph-mon[37215]: pgmap v137: 484 pgs: 38 creating+peering, 47 unknown, 399 active+clean; 217 MiB data, 1021 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-26T04:20:44.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:44 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:44.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:44 smithi016 ceph-mon[37215]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:20:44.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:44 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-11", "mode": "writeback"}]': finished 2024-03-26T04:20:44.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:44 smithi016 ceph-mon[37215]: from='client.40450 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi016-74578-10"}]': finished 2024-03-26T04:20:44.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:44 smithi016 ceph-mon[37215]: osdmap e120: 8 total, 8 up, 8 in 2024-03-26T04:20:44.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:44 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2129593090' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi016-74285-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:44.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:44 smithi016 ceph-mon[37215]: from='client.56102 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi016-74285-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:44.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:44 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/334241781' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi016-74578-15"}]: dispatch 2024-03-26T04:20:44.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:44 smithi016 ceph-mon[37215]: from='client.48637 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi016-74578-15"}]: dispatch 2024-03-26T04:20:44.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:44 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/334241781' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi016-74578-15"}]: dispatch 2024-03-26T04:20:44.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:44 smithi016 ceph-mon[37215]: from='client.48637 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi016-74578-15"}]: dispatch 2024-03-26T04:20:44.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:44 smithi183 ceph-mon[33745]: pgmap v137: 484 pgs: 38 creating+peering, 47 unknown, 399 active+clean; 217 MiB data, 1021 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-26T04:20:44.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:44 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:44.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:44 smithi183 ceph-mon[33745]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:20:44.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:44 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-11", "mode": "writeback"}]': finished 2024-03-26T04:20:44.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:44 smithi183 ceph-mon[33745]: from='client.40450 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsEC_smithi016-74578-10"}]': finished 2024-03-26T04:20:44.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:44 smithi183 ceph-mon[33745]: osdmap e120: 8 total, 8 up, 8 in 2024-03-26T04:20:44.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:44 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2129593090' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi016-74285-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:44.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:44 smithi183 ceph-mon[33745]: from='client.56102 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi016-74285-10","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:44.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:44 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/334241781' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi016-74578-15"}]: dispatch 2024-03-26T04:20:44.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:44 smithi183 ceph-mon[33745]: from='client.48637 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi016-74578-15"}]: dispatch 2024-03-26T04:20:44.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:44 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/334241781' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi016-74578-15"}]: dispatch 2024-03-26T04:20:44.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:44 smithi183 ceph-mon[33745]: from='client.48637 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi016-74578-15"}]: dispatch 2024-03-26T04:20:45.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:45 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/334241781' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi016-74578-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:45.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:45 smithi016 ceph-mon[29101]: from='client.48637 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi016-74578-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:45.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:45 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:45.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:45 smithi016 ceph-mon[29101]: from='client.56102 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi016-74285-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:45.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:45 smithi016 ceph-mon[29101]: from='client.48637 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi016-74578-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:20:45.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:45 smithi016 ceph-mon[29101]: osdmap e121: 8 total, 8 up, 8 in 2024-03-26T04:20:45.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:45 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/334241781' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi016-74578-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi016-74578-15"}]: dispatch 2024-03-26T04:20:45.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:45 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/970365944' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi016-74264-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:45.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:45 smithi016 ceph-mon[29101]: from='client.56111 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi016-74264-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:45.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:45 smithi016 ceph-mon[29101]: from='client.48637 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi016-74578-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi016-74578-15"}]: dispatch 2024-03-26T04:20:45.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:45 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/334241781' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi016-74578-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:45.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:45 smithi016 ceph-mon[37215]: from='client.48637 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi016-74578-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:45.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:45 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:45.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:45 smithi016 ceph-mon[37215]: from='client.56102 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi016-74285-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:45.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:45 smithi016 ceph-mon[37215]: from='client.48637 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi016-74578-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:20:45.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:45 smithi016 ceph-mon[37215]: osdmap e121: 8 total, 8 up, 8 in 2024-03-26T04:20:45.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:45 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/334241781' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi016-74578-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi016-74578-15"}]: dispatch 2024-03-26T04:20:45.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:45 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/970365944' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi016-74264-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:45.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:45 smithi016 ceph-mon[37215]: from='client.56111 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi016-74264-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:45.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:45 smithi016 ceph-mon[37215]: from='client.48637 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi016-74578-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi016-74578-15"}]: dispatch 2024-03-26T04:20:45.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:45 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/334241781' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi016-74578-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:45.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:45 smithi183 ceph-mon[33745]: from='client.48637 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi016-74578-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:20:45.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:45 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:45.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:45 smithi183 ceph-mon[33745]: from='client.56102 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripSparseReadPP_smithi016-74285-10","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:45.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:45 smithi183 ceph-mon[33745]: from='client.48637 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi016-74578-15", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:20:45.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:45 smithi183 ceph-mon[33745]: osdmap e121: 8 total, 8 up, 8 in 2024-03-26T04:20:45.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:45 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/334241781' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi016-74578-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi016-74578-15"}]: dispatch 2024-03-26T04:20:45.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:45 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/970365944' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi016-74264-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:45.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:45 smithi183 ceph-mon[33745]: from='client.56111 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi016-74264-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:45.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:45 smithi183 ceph-mon[33745]: from='client.48637 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi016-74578-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi016-74578-15"}]: dispatch 2024-03-26T04:20:46.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:46 smithi016 ceph-mon[37215]: pgmap v140: 484 pgs: 9 creating+peering, 87 unknown, 388 active+clean; 217 MiB data, 1021 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-26T04:20:46.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:46 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:20:46.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:46 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:46.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:46 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:20:46.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:46 smithi016 ceph-mon[37215]: from='client.56111 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi016-74264-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:46.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:46 smithi016 ceph-mon[29101]: pgmap v140: 484 pgs: 9 creating+peering, 87 unknown, 388 active+clean; 217 MiB data, 1021 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-26T04:20:46.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:46 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:20:46.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:46 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:46.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:46 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:20:46.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:46 smithi016 ceph-mon[29101]: from='client.56111 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi016-74264-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:46.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:46 smithi183 ceph-mon[33745]: pgmap v140: 484 pgs: 9 creating+peering, 87 unknown, 388 active+clean; 217 MiB data, 1021 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 2.0 KiB/s wr, 3 op/s 2024-03-26T04:20:46.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:46 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:20:46.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:46 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:46.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:46 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:20:46.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:46 smithi183 ceph-mon[33745]: from='client.56111 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValue_smithi016-74264-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:47.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:47 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:20:47.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:47 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-11"}]: dispatch 2024-03-26T04:20:47.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:47 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:47.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:47 smithi016 ceph-mon[29101]: osdmap e122: 8 total, 8 up, 8 in 2024-03-26T04:20:47.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:47 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-11"}]: dispatch 2024-03-26T04:20:47.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:47 smithi016 ceph-mon[29101]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:20:47.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:47 smithi016 ceph-mon[29101]: from='client.48637 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi016-74578-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi016-74578-15"}]': finished 2024-03-26T04:20:47.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:47 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-11"}]': finished 2024-03-26T04:20:47.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:47 smithi016 ceph-mon[29101]: osdmap e123: 8 total, 8 up, 8 in 2024-03-26T04:20:47.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:47 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3993284896' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi016-74285-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:47.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:47 smithi016 ceph-mon[29101]: from='client.56123 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi016-74285-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:47.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:47 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:20:47.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:47 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-11"}]: dispatch 2024-03-26T04:20:47.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:47 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:47.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:47 smithi016 ceph-mon[37215]: osdmap e122: 8 total, 8 up, 8 in 2024-03-26T04:20:47.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:47 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-11"}]: dispatch 2024-03-26T04:20:47.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:47 smithi016 ceph-mon[37215]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:20:47.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:47 smithi016 ceph-mon[37215]: from='client.48637 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi016-74578-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi016-74578-15"}]': finished 2024-03-26T04:20:47.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:47 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-11"}]': finished 2024-03-26T04:20:47.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:47 smithi016 ceph-mon[37215]: osdmap e123: 8 total, 8 up, 8 in 2024-03-26T04:20:47.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:47 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3993284896' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi016-74285-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:47.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:47 smithi016 ceph-mon[37215]: from='client.56123 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi016-74285-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:48.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:47 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:20:48.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:47 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-11"}]: dispatch 2024-03-26T04:20:48.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:47 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:48.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:47 smithi183 ceph-mon[33745]: osdmap e122: 8 total, 8 up, 8 in 2024-03-26T04:20:48.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:47 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-11"}]: dispatch 2024-03-26T04:20:48.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:47 smithi183 ceph-mon[33745]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:20:48.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:47 smithi183 ceph-mon[33745]: from='client.48637 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedEC_smithi016-74578-15", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedEC_smithi016-74578-15"}]': finished 2024-03-26T04:20:48.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:47 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-11"}]': finished 2024-03-26T04:20:48.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:47 smithi183 ceph-mon[33745]: osdmap e123: 8 total, 8 up, 8 in 2024-03-26T04:20:48.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:47 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3993284896' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi016-74285-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:48.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:47 smithi183 ceph-mon[33745]: from='client.56123 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi016-74285-11","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:48.076 INFO:tasks.workunit.client.0.smithi016.stdout: read_operations: Running main() from gmock_main.cc 2024-03-26T04:20:48.076 INFO:tasks.workunit.client.0.smithi016.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2024-03-26T04:20:48.076 INFO:tasks.workunit.client.0.smithi016.stdout: read_operations: [----------] Global test environment set-up. 2024-03-26T04:20:48.076 INFO:tasks.workunit.client.0.smithi016.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2024-03-26T04:20:48.076 INFO:tasks.workunit.client.0.smithi016.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2024-03-26T04:20:48.076 INFO:tasks.workunit.client.0.smithi016.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (10815 ms) 2024-03-26T04:20:48.077 INFO:tasks.workunit.client.0.smithi016.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2024-03-26T04:20:48.077 INFO:tasks.workunit.client.0.smithi016.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (5143 ms) 2024-03-26T04:20:48.077 INFO:tasks.workunit.client.0.smithi016.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2024-03-26T04:20:48.077 INFO:tasks.workunit.client.0.smithi016.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (4387 ms) 2024-03-26T04:20:48.077 INFO:tasks.workunit.client.0.smithi016.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2024-03-26T04:20:48.077 INFO:tasks.workunit.client.0.smithi016.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (3588 ms) 2024-03-26T04:20:48.077 INFO:tasks.workunit.client.0.smithi016.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2024-03-26T04:20:48.077 INFO:tasks.workunit.client.0.smithi016.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (5020 ms) 2024-03-26T04:20:48.077 INFO:tasks.workunit.client.0.smithi016.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2024-03-26T04:20:48.077 INFO:tasks.workunit.client.0.smithi016.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (3094 ms) 2024-03-26T04:20:48.077 INFO:tasks.workunit.client.0.smithi016.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2024-03-26T04:20:48.077 INFO:tasks.workunit.client.0.smithi016.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (3210 ms) 2024-03-26T04:20:48.077 INFO:tasks.workunit.client.0.smithi016.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2024-03-26T04:20:48.077 INFO:tasks.workunit.client.0.smithi016.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (3662 ms) 2024-03-26T04:20:48.078 INFO:tasks.workunit.client.0.smithi016.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2024-03-26T04:20:48.078 INFO:tasks.workunit.client.0.smithi016.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (3976 ms) 2024-03-26T04:20:48.078 INFO:tasks.workunit.client.0.smithi016.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2024-03-26T04:20:48.078 INFO:tasks.workunit.client.0.smithi016.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (3307 ms) 2024-03-26T04:20:48.078 INFO:tasks.workunit.client.0.smithi016.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2024-03-26T04:20:48.078 INFO:tasks.workunit.client.0.smithi016.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (3913 ms) 2024-03-26T04:20:48.078 INFO:tasks.workunit.client.0.smithi016.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2024-03-26T04:20:48.078 INFO:tasks.workunit.client.0.smithi016.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (3001 ms) 2024-03-26T04:20:48.078 INFO:tasks.workunit.client.0.smithi016.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2024-03-26T04:20:48.078 INFO:tasks.workunit.client.0.smithi016.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (3017 ms) 2024-03-26T04:20:48.078 INFO:tasks.workunit.client.0.smithi016.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2024-03-26T04:20:48.078 INFO:tasks.workunit.client.0.smithi016.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (2987 ms) 2024-03-26T04:20:48.078 INFO:tasks.workunit.client.0.smithi016.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (59120 ms total) 2024-03-26T04:20:48.079 INFO:tasks.workunit.client.0.smithi016.stdout: read_operations: 2024-03-26T04:20:48.079 INFO:tasks.workunit.client.0.smithi016.stdout: read_operations: [----------] Global test environment tear-down 2024-03-26T04:20:48.079 INFO:tasks.workunit.client.0.smithi016.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (59121 ms total) 2024-03-26T04:20:48.079 INFO:tasks.workunit.client.0.smithi016.stdout: read_operations: [ PASSED ] 14 tests. 2024-03-26T04:20:48.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:20:48 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:20:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:20:48.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:48 smithi016 ceph-mon[29101]: pgmap v143: 484 pgs: 13 creating+peering, 76 unknown, 395 active+clean; 217 MiB data, 1022 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:20:48.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:48 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:48.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:48 smithi016 ceph-mon[29101]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:20:48.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:48 smithi016 ceph-mon[29101]: from='client.56123 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi016-74285-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:48.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:48 smithi016 ceph-mon[29101]: osdmap e124: 8 total, 8 up, 8 in 2024-03-26T04:20:48.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:48 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1394250086' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi016-74264-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:48.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:48 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:48.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:48 smithi016 ceph-mon[29101]: from='client.48664 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi016-74264-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:48.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:48 smithi016 ceph-mon[37215]: pgmap v143: 484 pgs: 13 creating+peering, 76 unknown, 395 active+clean; 217 MiB data, 1022 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:20:48.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:48 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:48.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:48 smithi016 ceph-mon[37215]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:20:48.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:48 smithi016 ceph-mon[37215]: from='client.56123 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi016-74285-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:48.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:48 smithi016 ceph-mon[37215]: osdmap e124: 8 total, 8 up, 8 in 2024-03-26T04:20:48.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:48 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1394250086' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi016-74264-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:48.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:48 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:48.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:48 smithi016 ceph-mon[37215]: from='client.48664 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi016-74264-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:49.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:48 smithi183 ceph-mon[33745]: pgmap v143: 484 pgs: 13 creating+peering, 76 unknown, 395 active+clean; 217 MiB data, 1022 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:20:49.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:48 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:49.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:48 smithi183 ceph-mon[33745]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:20:49.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:48 smithi183 ceph-mon[33745]: from='client.56123 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsCompletePP_smithi016-74285-11","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:49.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:48 smithi183 ceph-mon[33745]: osdmap e124: 8 total, 8 up, 8 in 2024-03-26T04:20:49.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:48 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1394250086' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi016-74264-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:49.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:48 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:49.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:48 smithi183 ceph-mon[33745]: from='client.48664 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Flush_smithi016-74264-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:50.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:50 smithi016 ceph-mon[37215]: pgmap v146: 460 pgs: 14 creating+peering, 90 unknown, 356 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-26T04:20:50.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:50 smithi016 ceph-mon[37215]: from='client.48664 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi016-74264-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:50.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:50 smithi016 ceph-mon[37215]: osdmap e125: 8 total, 8 up, 8 in 2024-03-26T04:20:50.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:50 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:50.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:50 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:50.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:50 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:50.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:50 smithi016 ceph-mon[29101]: pgmap v146: 460 pgs: 14 creating+peering, 90 unknown, 356 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-26T04:20:50.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:50 smithi016 ceph-mon[29101]: from='client.48664 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi016-74264-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:50.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:50 smithi016 ceph-mon[29101]: osdmap e125: 8 total, 8 up, 8 in 2024-03-26T04:20:50.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:50 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:50.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:50 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:50.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:50 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:50.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:50 smithi183 ceph-mon[33745]: pgmap v146: 460 pgs: 14 creating+peering, 90 unknown, 356 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-26T04:20:50.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:50 smithi183 ceph-mon[33745]: from='client.48664 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Flush_smithi016-74264-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:50.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:50 smithi183 ceph-mon[33745]: osdmap e125: 8 total, 8 up, 8 in 2024-03-26T04:20:50.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:50 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:50.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:50 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:50.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:50 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:51.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:51 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:51.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:51 smithi016 ceph-mon[37215]: osdmap e126: 8 total, 8 up, 8 in 2024-03-26T04:20:51.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:51 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3733643468' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi016-74285-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:51.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:51 smithi016 ceph-mon[37215]: from='client.48670 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi016-74285-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:51.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:51 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:51.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:51 smithi016 ceph-mon[37215]: from='client.48670 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi016-74285-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:51.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:51 smithi016 ceph-mon[37215]: osdmap e127: 8 total, 8 up, 8 in 2024-03-26T04:20:51.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:51 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2468469771' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi016-74264-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:51.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:51 smithi016 ceph-mon[37215]: from='client.56144 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi016-74264-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:51.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:51 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:51.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:51 smithi016 ceph-mon[29101]: osdmap e126: 8 total, 8 up, 8 in 2024-03-26T04:20:51.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:51 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3733643468' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi016-74285-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:51.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:51 smithi016 ceph-mon[29101]: from='client.48670 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi016-74285-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:51.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:51 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:51.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:51 smithi016 ceph-mon[29101]: from='client.48670 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi016-74285-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:51.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:51 smithi016 ceph-mon[29101]: osdmap e127: 8 total, 8 up, 8 in 2024-03-26T04:20:51.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:51 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2468469771' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi016-74264-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:51.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:51 smithi016 ceph-mon[29101]: from='client.56144 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi016-74264-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:51.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:51 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:51.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:51 smithi183 ceph-mon[33745]: osdmap e126: 8 total, 8 up, 8 in 2024-03-26T04:20:51.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:51 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3733643468' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi016-74285-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:51.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:51 smithi183 ceph-mon[33745]: from='client.48670 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi016-74285-12","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:51.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:51 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:51.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:51 smithi183 ceph-mon[33745]: from='client.48670 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "IsSafePP_smithi016-74285-12","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:51.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:51 smithi183 ceph-mon[33745]: osdmap e127: 8 total, 8 up, 8 in 2024-03-26T04:20:51.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:51 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2468469771' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi016-74264-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:51.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:51 smithi183 ceph-mon[33745]: from='client.56144 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi016-74264-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:52.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:52 smithi016 ceph-mon[29101]: pgmap v149: 460 pgs: 3 creating+peering, 101 unknown, 356 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-26T04:20:52.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:52 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:52.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:52 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:20:52.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:52 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:20:52.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:52 smithi016 ceph-mon[29101]: from='client.56144 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi016-74264-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:52.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:52 smithi016 ceph-mon[29101]: osdmap e128: 8 total, 8 up, 8 in 2024-03-26T04:20:52.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:52 smithi016 ceph-mon[37215]: pgmap v149: 460 pgs: 3 creating+peering, 101 unknown, 356 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-26T04:20:52.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:52 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:52.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:52 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:20:52.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:52 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:20:52.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:52 smithi016 ceph-mon[37215]: from='client.56144 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi016-74264-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:52.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:52 smithi016 ceph-mon[37215]: osdmap e128: 8 total, 8 up, 8 in 2024-03-26T04:20:52.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:20:52 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:20:52.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:52 smithi183 ceph-mon[33745]: pgmap v149: 460 pgs: 3 creating+peering, 101 unknown, 356 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-26T04:20:52.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:52 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:52.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:52 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:20:52.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:52 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:20:52.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:52 smithi183 ceph-mon[33745]: from='client.56144 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsync_smithi016-74264-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:52.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:52 smithi183 ceph-mon[33745]: osdmap e128: 8 total, 8 up, 8 in 2024-03-26T04:20:53.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:53 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:20:53.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:53 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:53.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:53 smithi016 ceph-mon[29101]: osdmap e129: 8 total, 8 up, 8 in 2024-03-26T04:20:53.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:53 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3238580671' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi016-74285-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:53.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:53 smithi016 ceph-mon[29101]: from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi016-74285-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:53.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:53 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:20:53.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:53 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:53.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:53 smithi016 ceph-mon[37215]: osdmap e129: 8 total, 8 up, 8 in 2024-03-26T04:20:53.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:53 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3238580671' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi016-74285-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:53.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:53 smithi016 ceph-mon[37215]: from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi016-74285-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:53.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:53 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:20:53.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:53 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:53.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:53 smithi183 ceph-mon[33745]: osdmap e129: 8 total, 8 up, 8 in 2024-03-26T04:20:53.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:53 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3238580671' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi016-74285-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:53.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:53 smithi183 ceph-mon[33745]: from='client.48682 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi016-74285-13","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:54.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:54 smithi016 ceph-mon[37215]: pgmap v152: 460 pgs: 23 creating+peering, 437 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 60 KiB/s wr, 59 op/s 2024-03-26T04:20:54.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:54 smithi016 ceph-mon[37215]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:20:54.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:54 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:54.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:54 smithi016 ceph-mon[37215]: from='client.48682 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi016-74285-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:54.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:54 smithi016 ceph-mon[37215]: osdmap e130: 8 total, 8 up, 8 in 2024-03-26T04:20:54.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:54 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/4093398643' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi016-74264-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:54.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:54 smithi016 ceph-mon[37215]: from='client.48688 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi016-74264-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:54.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:54 smithi016 ceph-mon[29101]: pgmap v152: 460 pgs: 23 creating+peering, 437 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 60 KiB/s wr, 59 op/s 2024-03-26T04:20:54.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:54 smithi016 ceph-mon[29101]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:20:54.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:54 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:54.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:54 smithi016 ceph-mon[29101]: from='client.48682 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi016-74285-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:54.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:54 smithi016 ceph-mon[29101]: osdmap e130: 8 total, 8 up, 8 in 2024-03-26T04:20:54.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:54 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/4093398643' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi016-74264-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:54.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:54 smithi016 ceph-mon[29101]: from='client.48688 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi016-74264-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:54.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:54 smithi183 ceph-mon[33745]: pgmap v152: 460 pgs: 23 creating+peering, 437 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 60 KiB/s wr, 59 op/s 2024-03-26T04:20:54.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:54 smithi183 ceph-mon[33745]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:20:54.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:54 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:54.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:54 smithi183 ceph-mon[33745]: from='client.48682 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReturnValuePP_smithi016-74285-13","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:54.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:54 smithi183 ceph-mon[33745]: osdmap e130: 8 total, 8 up, 8 in 2024-03-26T04:20:54.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:54 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/4093398643' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi016-74264-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:54.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:54 smithi183 ceph-mon[33745]: from='client.48688 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi016-74264-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:55.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:55 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:55.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:55 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:20:55.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:55 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:20:55.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:55 smithi016 ceph-mon[37215]: from='client.48688 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi016-74264-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:55.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:55 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-13", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:20:55.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:55 smithi016 ceph-mon[37215]: osdmap e131: 8 total, 8 up, 8 in 2024-03-26T04:20:55.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:55 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-13"}]: dispatch 2024-03-26T04:20:55.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:55 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-13"}]: dispatch 2024-03-26T04:20:55.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:55 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:55.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:55 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:20:55.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:55 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:20:55.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:55 smithi016 ceph-mon[29101]: from='client.48688 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi016-74264-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:55.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:55 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-13", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:20:55.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:55 smithi016 ceph-mon[29101]: osdmap e131: 8 total, 8 up, 8 in 2024-03-26T04:20:55.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:55 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-13"}]: dispatch 2024-03-26T04:20:55.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:55 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-13"}]: dispatch 2024-03-26T04:20:55.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:55 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:55.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:55 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:20:55.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:55 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-13", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:20:55.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:55 smithi183 ceph-mon[33745]: from='client.48688 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFull_smithi016-74264-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:55.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:55 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-13", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:20:55.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:55 smithi183 ceph-mon[33745]: osdmap e131: 8 total, 8 up, 8 in 2024-03-26T04:20:55.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:55 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-13"}]: dispatch 2024-03-26T04:20:55.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:55 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-13"}]: dispatch 2024-03-26T04:20:56.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:56 smithi016 ceph-mon[29101]: pgmap v155: 492 pgs: 64 unknown, 428 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 60 KiB/s wr, 59 op/s 2024-03-26T04:20:56.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:56 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:56.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:56 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-13"}]': finished 2024-03-26T04:20:56.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:56 smithi016 ceph-mon[29101]: osdmap e132: 8 total, 8 up, 8 in 2024-03-26T04:20:56.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:56 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-13", "mode": "writeback"}]: dispatch 2024-03-26T04:20:56.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:56 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/334241781' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi016-74578-15"}]: dispatch 2024-03-26T04:20:56.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:56 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-13", "mode": "writeback"}]: dispatch 2024-03-26T04:20:56.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:56 smithi016 ceph-mon[29101]: from='client.48637 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi016-74578-15"}]: dispatch 2024-03-26T04:20:56.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:56 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2045543960' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi016-74285-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:56.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:56 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:56.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:56 smithi016 ceph-mon[37215]: pgmap v155: 492 pgs: 64 unknown, 428 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 60 KiB/s wr, 59 op/s 2024-03-26T04:20:56.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:56 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:56.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:56 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-13"}]': finished 2024-03-26T04:20:56.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:56 smithi016 ceph-mon[37215]: osdmap e132: 8 total, 8 up, 8 in 2024-03-26T04:20:56.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:56 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-13", "mode": "writeback"}]: dispatch 2024-03-26T04:20:56.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:56 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/334241781' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi016-74578-15"}]: dispatch 2024-03-26T04:20:56.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:56 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-13", "mode": "writeback"}]: dispatch 2024-03-26T04:20:56.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:56 smithi016 ceph-mon[37215]: from='client.48637 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi016-74578-15"}]: dispatch 2024-03-26T04:20:56.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:56 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2045543960' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi016-74285-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:56.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:56 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:56.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:56 smithi183 ceph-mon[33745]: pgmap v155: 492 pgs: 64 unknown, 428 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 60 KiB/s wr, 59 op/s 2024-03-26T04:20:56.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:56 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:56.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:56 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-13"}]': finished 2024-03-26T04:20:56.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:56 smithi183 ceph-mon[33745]: osdmap e132: 8 total, 8 up, 8 in 2024-03-26T04:20:56.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:56 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-13", "mode": "writeback"}]: dispatch 2024-03-26T04:20:56.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:56 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/334241781' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi016-74578-15"}]: dispatch 2024-03-26T04:20:56.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:56 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-13", "mode": "writeback"}]: dispatch 2024-03-26T04:20:56.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:56 smithi183 ceph-mon[33745]: from='client.48637 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi016-74578-15"}]: dispatch 2024-03-26T04:20:56.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:56 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2045543960' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushPP_smithi016-74285-14","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:56.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:56 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:57.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:57 smithi016 ceph-mon[37215]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:20:57.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:57 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-13", "mode": "writeback"}]': finished 2024-03-26T04:20:57.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:57 smithi016 ceph-mon[37215]: from='client.48637 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi016-74578-15"}]': finished 2024-03-26T04:20:57.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:57 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2045543960' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi016-74285-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:57.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:57 smithi016 ceph-mon[37215]: osdmap e133: 8 total, 8 up, 8 in 2024-03-26T04:20:57.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:57 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/734104665' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi016-74264-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:57.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:57 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/334241781' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi016-74578-15"}]: dispatch 2024-03-26T04:20:57.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:57 smithi016 ceph-mon[37215]: from='client.56168 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi016-74264-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:57.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:57 smithi016 ceph-mon[37215]: from='client.48637 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi016-74578-15"}]: dispatch 2024-03-26T04:20:57.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:57 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:57.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:57 smithi016 ceph-mon[29101]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:20:57.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:57 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-13", "mode": "writeback"}]': finished 2024-03-26T04:20:57.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:57 smithi016 ceph-mon[29101]: from='client.48637 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi016-74578-15"}]': finished 2024-03-26T04:20:57.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:57 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2045543960' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi016-74285-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:57.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:57 smithi016 ceph-mon[29101]: osdmap e133: 8 total, 8 up, 8 in 2024-03-26T04:20:57.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:57 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/734104665' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi016-74264-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:57.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:57 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/334241781' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi016-74578-15"}]: dispatch 2024-03-26T04:20:57.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:57 smithi016 ceph-mon[29101]: from='client.56168 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi016-74264-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:57.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:57 smithi016 ceph-mon[29101]: from='client.48637 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi016-74578-15"}]: dispatch 2024-03-26T04:20:57.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:57 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:57.692 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:57 smithi183 ceph-mon[33745]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:20:57.692 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:57 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-13", "mode": "writeback"}]': finished 2024-03-26T04:20:57.692 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:57 smithi183 ceph-mon[33745]: from='client.48637 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedEC_smithi016-74578-15"}]': finished 2024-03-26T04:20:57.692 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:57 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2045543960' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushPP_smithi016-74285-14","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:57.692 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:57 smithi183 ceph-mon[33745]: osdmap e133: 8 total, 8 up, 8 in 2024-03-26T04:20:57.693 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:57 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/734104665' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi016-74264-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:57.693 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:57 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/334241781' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi016-74578-15"}]: dispatch 2024-03-26T04:20:57.693 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:57 smithi183 ceph-mon[33745]: from='client.56168 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi016-74264-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:57.693 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:57 smithi183 ceph-mon[33745]: from='client.48637 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi016-74578-15"}]: dispatch 2024-03-26T04:20:57.693 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:57 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:58.075 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots: Running main() from gmock_main.cc 2024-03-26T04:20:58.075 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-03-26T04:20:58.075 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots: [----------] Global test environment set-up. 2024-03-26T04:20:58.075 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-03-26T04:20:58.076 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-03-26T04:20:58.076 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (2378 ms) 2024-03-26T04:20:58.076 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-03-26T04:20:58.076 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (2185 ms) 2024-03-26T04:20:58.076 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-03-26T04:20:58.076 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (2111 ms) 2024-03-26T04:20:58.076 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-03-26T04:20:58.076 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (2061 ms) 2024-03-26T04:20:58.076 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (8735 ms total) 2024-03-26T04:20:58.076 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots: 2024-03-26T04:20:58.076 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-03-26T04:20:58.076 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-03-26T04:20:58.076 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (4733 ms) 2024-03-26T04:20:58.076 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-03-26T04:20:58.077 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (4515 ms) 2024-03-26T04:20:58.077 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-03-26T04:20:58.077 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (5815 ms) 2024-03-26T04:20:58.077 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (15063 ms total) 2024-03-26T04:20:58.077 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots: 2024-03-26T04:20:58.077 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-03-26T04:20:58.077 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-03-26T04:20:58.077 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (3267 ms) 2024-03-26T04:20:58.077 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-03-26T04:20:58.077 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (1948 ms) 2024-03-26T04:20:58.077 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-03-26T04:20:58.077 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (2004 ms) 2024-03-26T04:20:58.077 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-03-26T04:20:58.077 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (1998 ms) 2024-03-26T04:20:58.078 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (9218 ms total) 2024-03-26T04:20:58.078 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots: 2024-03-26T04:20:58.078 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-03-26T04:20:58.078 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-03-26T04:20:58.078 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (4034 ms) 2024-03-26T04:20:58.078 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-03-26T04:20:58.078 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (4006 ms) 2024-03-26T04:20:58.078 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (8040 ms total) 2024-03-26T04:20:58.078 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots: 2024-03-26T04:20:58.078 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots: [----------] Global test environment tear-down 2024-03-26T04:20:58.078 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (69449 ms total) 2024-03-26T04:20:58.078 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-03-26T04:20:58.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:58 smithi016 ceph-mon[29101]: pgmap v158: 452 pgs: 13 creating+peering, 51 unknown, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 0 B/s wr, 2 op/s 2024-03-26T04:20:58.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.0"}]: dispatch 2024-03-26T04:20:58.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.1"}]: dispatch 2024-03-26T04:20:58.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.2"}]: dispatch 2024-03-26T04:20:58.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.3"}]: dispatch 2024-03-26T04:20:58.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.4"}]: dispatch 2024-03-26T04:20:58.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.5"}]: dispatch 2024-03-26T04:20:58.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.6"}]: dispatch 2024-03-26T04:20:58.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.7"}]: dispatch 2024-03-26T04:20:58.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.8"}]: dispatch 2024-03-26T04:20:58.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.9"}]: dispatch 2024-03-26T04:20:58.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:58 smithi016 ceph-mon[29101]: 161.9 scrub starts 2024-03-26T04:20:58.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:58 smithi016 ceph-mon[29101]: 161.9 scrub ok 2024-03-26T04:20:58.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:58 smithi016 ceph-mon[29101]: from='client.56168 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi016-74264-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:58.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:58 smithi016 ceph-mon[29101]: from='client.48637 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi016-74578-15"}]': finished 2024-03-26T04:20:58.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:58 smithi016 ceph-mon[29101]: osdmap e134: 8 total, 8 up, 8 in 2024-03-26T04:20:58.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:58 smithi016 ceph-mon[29101]: 161.8 scrub starts 2024-03-26T04:20:58.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:58 smithi016 ceph-mon[29101]: 161.8 scrub ok 2024-03-26T04:20:58.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:58.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:58 smithi016 ceph-mon[37215]: pgmap v158: 452 pgs: 13 creating+peering, 51 unknown, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 0 B/s wr, 2 op/s 2024-03-26T04:20:58.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.0"}]: dispatch 2024-03-26T04:20:58.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.1"}]: dispatch 2024-03-26T04:20:58.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.2"}]: dispatch 2024-03-26T04:20:58.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.3"}]: dispatch 2024-03-26T04:20:58.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.4"}]: dispatch 2024-03-26T04:20:58.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.5"}]: dispatch 2024-03-26T04:20:58.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.6"}]: dispatch 2024-03-26T04:20:58.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.7"}]: dispatch 2024-03-26T04:20:58.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.8"}]: dispatch 2024-03-26T04:20:58.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.9"}]: dispatch 2024-03-26T04:20:58.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:58 smithi016 ceph-mon[37215]: 161.9 scrub starts 2024-03-26T04:20:58.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:58 smithi016 ceph-mon[37215]: 161.9 scrub ok 2024-03-26T04:20:58.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:58 smithi016 ceph-mon[37215]: from='client.56168 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi016-74264-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:58.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:58 smithi016 ceph-mon[37215]: from='client.48637 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi016-74578-15"}]': finished 2024-03-26T04:20:58.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:58 smithi016 ceph-mon[37215]: osdmap e134: 8 total, 8 up, 8 in 2024-03-26T04:20:58.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:58 smithi016 ceph-mon[37215]: 161.8 scrub starts 2024-03-26T04:20:58.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:58 smithi016 ceph-mon[37215]: 161.8 scrub ok 2024-03-26T04:20:58.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:58.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:58 smithi183 ceph-mon[33745]: pgmap v158: 452 pgs: 13 creating+peering, 51 unknown, 388 active+clean; 217 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 0 B/s wr, 2 op/s 2024-03-26T04:20:58.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.0"}]: dispatch 2024-03-26T04:20:58.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.1"}]: dispatch 2024-03-26T04:20:58.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.2"}]: dispatch 2024-03-26T04:20:58.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.3"}]: dispatch 2024-03-26T04:20:58.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.4"}]: dispatch 2024-03-26T04:20:58.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.5"}]: dispatch 2024-03-26T04:20:58.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.6"}]: dispatch 2024-03-26T04:20:58.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.7"}]: dispatch 2024-03-26T04:20:58.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.8"}]: dispatch 2024-03-26T04:20:58.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "161.9"}]: dispatch 2024-03-26T04:20:58.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:58 smithi183 ceph-mon[33745]: 161.9 scrub starts 2024-03-26T04:20:58.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:58 smithi183 ceph-mon[33745]: 161.9 scrub ok 2024-03-26T04:20:58.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:58 smithi183 ceph-mon[33745]: from='client.56168 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSame_smithi016-74264-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:20:58.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:58 smithi183 ceph-mon[33745]: from='client.48637 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedEC_smithi016-74578-15"}]': finished 2024-03-26T04:20:58.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:58 smithi183 ceph-mon[33745]: osdmap e134: 8 total, 8 up, 8 in 2024-03-26T04:20:58.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:58 smithi183 ceph-mon[33745]: 161.8 scrub starts 2024-03-26T04:20:58.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:58 smithi183 ceph-mon[33745]: 161.8 scrub ok 2024-03-26T04:20:58.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:59.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:20:58 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:20:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:20:59.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:59 smithi016 ceph-mon[29101]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.0"}]: dispatch 2024-03-26T04:20:59.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:59 smithi016 ceph-mon[29101]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.1"}]: dispatch 2024-03-26T04:20:59.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:59 smithi016 ceph-mon[29101]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.2"}]: dispatch 2024-03-26T04:20:59.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:59 smithi016 ceph-mon[29101]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.3"}]: dispatch 2024-03-26T04:20:59.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:59 smithi016 ceph-mon[29101]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.4"}]: dispatch 2024-03-26T04:20:59.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:59 smithi016 ceph-mon[29101]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.5"}]: dispatch 2024-03-26T04:20:59.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:59 smithi016 ceph-mon[29101]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.6"}]: dispatch 2024-03-26T04:20:59.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:59 smithi016 ceph-mon[29101]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.7"}]: dispatch 2024-03-26T04:20:59.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:59 smithi016 ceph-mon[29101]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.8"}]: dispatch 2024-03-26T04:20:59.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:59 smithi016 ceph-mon[29101]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.9"}]: dispatch 2024-03-26T04:20:59.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:59 smithi016 ceph-mon[29101]: 161.5 scrub starts 2024-03-26T04:20:59.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:59 smithi016 ceph-mon[29101]: 161.5 scrub ok 2024-03-26T04:20:59.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:59 smithi016 ceph-mon[29101]: osdmap e135: 8 total, 8 up, 8 in 2024-03-26T04:20:59.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:59 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1175157784' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi016-74285-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:59.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:59 smithi016 ceph-mon[29101]: from='client.48709 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi016-74285-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:59.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:20:59 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:20:59.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:59 smithi016 ceph-mon[37215]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.0"}]: dispatch 2024-03-26T04:20:59.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:59 smithi016 ceph-mon[37215]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.1"}]: dispatch 2024-03-26T04:20:59.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:59 smithi016 ceph-mon[37215]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.2"}]: dispatch 2024-03-26T04:20:59.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:59 smithi016 ceph-mon[37215]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.3"}]: dispatch 2024-03-26T04:20:59.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:59 smithi016 ceph-mon[37215]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.4"}]: dispatch 2024-03-26T04:20:59.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:59 smithi016 ceph-mon[37215]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.5"}]: dispatch 2024-03-26T04:20:59.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:59 smithi016 ceph-mon[37215]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.6"}]: dispatch 2024-03-26T04:20:59.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:59 smithi016 ceph-mon[37215]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.7"}]: dispatch 2024-03-26T04:20:59.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:59 smithi016 ceph-mon[37215]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.8"}]: dispatch 2024-03-26T04:20:59.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:59 smithi016 ceph-mon[37215]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.9"}]: dispatch 2024-03-26T04:20:59.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:59 smithi016 ceph-mon[37215]: 161.5 scrub starts 2024-03-26T04:20:59.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:59 smithi016 ceph-mon[37215]: 161.5 scrub ok 2024-03-26T04:20:59.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:59 smithi016 ceph-mon[37215]: osdmap e135: 8 total, 8 up, 8 in 2024-03-26T04:20:59.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:59 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1175157784' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi016-74285-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:59.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:59 smithi016 ceph-mon[37215]: from='client.48709 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi016-74285-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:20:59.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:20:59 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:00.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:59 smithi183 ceph-mon[33745]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.0"}]: dispatch 2024-03-26T04:21:00.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:59 smithi183 ceph-mon[33745]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.1"}]: dispatch 2024-03-26T04:21:00.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:59 smithi183 ceph-mon[33745]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.2"}]: dispatch 2024-03-26T04:21:00.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:59 smithi183 ceph-mon[33745]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.3"}]: dispatch 2024-03-26T04:21:00.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:59 smithi183 ceph-mon[33745]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.4"}]: dispatch 2024-03-26T04:21:00.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:59 smithi183 ceph-mon[33745]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.5"}]: dispatch 2024-03-26T04:21:00.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:59 smithi183 ceph-mon[33745]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.6"}]: dispatch 2024-03-26T04:21:00.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:59 smithi183 ceph-mon[33745]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.7"}]: dispatch 2024-03-26T04:21:00.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:59 smithi183 ceph-mon[33745]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.8"}]: dispatch 2024-03-26T04:21:00.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:59 smithi183 ceph-mon[33745]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "161.9"}]: dispatch 2024-03-26T04:21:00.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:59 smithi183 ceph-mon[33745]: 161.5 scrub starts 2024-03-26T04:21:00.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:59 smithi183 ceph-mon[33745]: 161.5 scrub ok 2024-03-26T04:21:00.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:59 smithi183 ceph-mon[33745]: osdmap e135: 8 total, 8 up, 8 in 2024-03-26T04:21:00.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:59 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1175157784' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi016-74285-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:00.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:59 smithi183 ceph-mon[33745]: from='client.48709 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi016-74285-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:00.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:20:59 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:00.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:00 smithi016 ceph-mon[29101]: 161.3 deep-scrub starts 2024-03-26T04:21:00.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:00 smithi016 ceph-mon[29101]: 161.3 deep-scrub ok 2024-03-26T04:21:00.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:00 smithi016 ceph-mon[29101]: 161.1 scrub starts 2024-03-26T04:21:00.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:00 smithi016 ceph-mon[29101]: 161.1 scrub ok 2024-03-26T04:21:00.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:00 smithi016 ceph-mon[29101]: 161.6 scrub starts 2024-03-26T04:21:00.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:00 smithi016 ceph-mon[29101]: 161.6 scrub ok 2024-03-26T04:21:00.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:00 smithi016 ceph-mon[29101]: 161.7 scrub starts 2024-03-26T04:21:00.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:00 smithi016 ceph-mon[29101]: 161.7 scrub ok 2024-03-26T04:21:00.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:00 smithi016 ceph-mon[29101]: pgmap v161: 452 pgs: 39 creating+peering, 16 unknown, 397 active+clean; 218 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 320 KiB/s wr, 70 op/s 2024-03-26T04:21:00.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:00 smithi016 ceph-mon[29101]: 161.4 deep-scrub starts 2024-03-26T04:21:00.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:00 smithi016 ceph-mon[29101]: 161.4 deep-scrub ok 2024-03-26T04:21:00.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:00 smithi016 ceph-mon[29101]: 161.0 scrub starts 2024-03-26T04:21:00.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:00 smithi016 ceph-mon[29101]: 161.0 scrub ok 2024-03-26T04:21:00.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:00 smithi016 ceph-mon[29101]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:21:00.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:00 smithi016 ceph-mon[29101]: from='client.48709 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi016-74285-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:00.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:00 smithi016 ceph-mon[29101]: osdmap e136: 8 total, 8 up, 8 in 2024-03-26T04:21:00.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:00 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1927152313' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74609-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:00.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:00 smithi016 ceph-mon[29101]: from='client.48715 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74609-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:00.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:00 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2015334992' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi016-74264-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:00.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:00 smithi016 ceph-mon[29101]: from='client.56174 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi016-74264-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:00.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:00 smithi016 ceph-mon[29101]: 161.2 deep-scrub starts 2024-03-26T04:21:00.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:00 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:00.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:00 smithi016 ceph-mon[37215]: 161.3 deep-scrub starts 2024-03-26T04:21:00.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:00 smithi016 ceph-mon[37215]: 161.3 deep-scrub ok 2024-03-26T04:21:00.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:00 smithi016 ceph-mon[37215]: 161.1 scrub starts 2024-03-26T04:21:00.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:00 smithi016 ceph-mon[37215]: 161.1 scrub ok 2024-03-26T04:21:00.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:00 smithi016 ceph-mon[37215]: 161.6 scrub starts 2024-03-26T04:21:00.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:00 smithi016 ceph-mon[37215]: 161.6 scrub ok 2024-03-26T04:21:00.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:00 smithi016 ceph-mon[37215]: 161.7 scrub starts 2024-03-26T04:21:00.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:00 smithi016 ceph-mon[37215]: 161.7 scrub ok 2024-03-26T04:21:00.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:00 smithi016 ceph-mon[37215]: pgmap v161: 452 pgs: 39 creating+peering, 16 unknown, 397 active+clean; 218 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 320 KiB/s wr, 70 op/s 2024-03-26T04:21:00.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:00 smithi016 ceph-mon[37215]: 161.4 deep-scrub starts 2024-03-26T04:21:00.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:00 smithi016 ceph-mon[37215]: 161.4 deep-scrub ok 2024-03-26T04:21:00.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:00 smithi016 ceph-mon[37215]: 161.0 scrub starts 2024-03-26T04:21:00.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:00 smithi016 ceph-mon[37215]: 161.0 scrub ok 2024-03-26T04:21:00.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:00 smithi016 ceph-mon[37215]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:21:00.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:00 smithi016 ceph-mon[37215]: from='client.48709 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi016-74285-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:00.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:00 smithi016 ceph-mon[37215]: osdmap e136: 8 total, 8 up, 8 in 2024-03-26T04:21:00.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:00 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1927152313' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74609-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:00.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:00 smithi016 ceph-mon[37215]: from='client.48715 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74609-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:00.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:00 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2015334992' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi016-74264-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:00.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:00 smithi016 ceph-mon[37215]: from='client.56174 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi016-74264-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:00.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:00 smithi016 ceph-mon[37215]: 161.2 deep-scrub starts 2024-03-26T04:21:00.916 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:00 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:01.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:00 smithi183 ceph-mon[33745]: 161.3 deep-scrub starts 2024-03-26T04:21:01.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:00 smithi183 ceph-mon[33745]: 161.3 deep-scrub ok 2024-03-26T04:21:01.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:00 smithi183 ceph-mon[33745]: 161.1 scrub starts 2024-03-26T04:21:01.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:00 smithi183 ceph-mon[33745]: 161.1 scrub ok 2024-03-26T04:21:01.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:00 smithi183 ceph-mon[33745]: 161.6 scrub starts 2024-03-26T04:21:01.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:00 smithi183 ceph-mon[33745]: 161.6 scrub ok 2024-03-26T04:21:01.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:00 smithi183 ceph-mon[33745]: 161.7 scrub starts 2024-03-26T04:21:01.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:00 smithi183 ceph-mon[33745]: 161.7 scrub ok 2024-03-26T04:21:01.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:00 smithi183 ceph-mon[33745]: pgmap v161: 452 pgs: 39 creating+peering, 16 unknown, 397 active+clean; 218 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 320 KiB/s wr, 70 op/s 2024-03-26T04:21:01.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:00 smithi183 ceph-mon[33745]: 161.4 deep-scrub starts 2024-03-26T04:21:01.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:00 smithi183 ceph-mon[33745]: 161.4 deep-scrub ok 2024-03-26T04:21:01.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:00 smithi183 ceph-mon[33745]: 161.0 scrub starts 2024-03-26T04:21:01.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:00 smithi183 ceph-mon[33745]: 161.0 scrub ok 2024-03-26T04:21:01.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:00 smithi183 ceph-mon[33745]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:21:01.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:00 smithi183 ceph-mon[33745]: from='client.48709 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "FlushAsyncPP_smithi016-74285-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:01.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:00 smithi183 ceph-mon[33745]: osdmap e136: 8 total, 8 up, 8 in 2024-03-26T04:21:01.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:00 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1927152313' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74609-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:01.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:00 smithi183 ceph-mon[33745]: from='client.48715 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74609-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:01.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:00 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2015334992' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi016-74264-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:01.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:00 smithi183 ceph-mon[33745]: from='client.56174 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi016-74264-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:01.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:00 smithi183 ceph-mon[33745]: 161.2 deep-scrub starts 2024-03-26T04:21:01.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:00 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:02.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:02 smithi016 ceph-mon[37215]: 161.2 deep-scrub ok 2024-03-26T04:21:02.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:02 smithi016 ceph-mon[37215]: pgmap v164: 484 pgs: 23 creating+peering, 96 unknown, 365 active+clean; 218 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 59 KiB/s wr, 59 op/s 2024-03-26T04:21:02.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:02 smithi016 ceph-mon[37215]: from='client.48715 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74609-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:02.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:02 smithi016 ceph-mon[37215]: from='client.56174 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi016-74264-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:02.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:02 smithi016 ceph-mon[37215]: osdmap e137: 8 total, 8 up, 8 in 2024-03-26T04:21:02.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:02 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:02.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:02 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-26T04:21:02.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:02 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi016-74514-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:02.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:02 smithi016 ceph-mon[37215]: from='client.24733 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi016-74514-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:02.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:02 smithi016 ceph-mon[29101]: 161.2 deep-scrub ok 2024-03-26T04:21:02.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:02 smithi016 ceph-mon[29101]: pgmap v164: 484 pgs: 23 creating+peering, 96 unknown, 365 active+clean; 218 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 59 KiB/s wr, 59 op/s 2024-03-26T04:21:02.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:02 smithi016 ceph-mon[29101]: from='client.48715 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74609-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:02.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:02 smithi016 ceph-mon[29101]: from='client.56174 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi016-74264-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:02.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:02 smithi016 ceph-mon[29101]: osdmap e137: 8 total, 8 up, 8 in 2024-03-26T04:21:02.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:02 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:02.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:02 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-26T04:21:02.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:02 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi016-74514-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:02.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:02 smithi016 ceph-mon[29101]: from='client.24733 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi016-74514-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:02.542 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:21:02 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:21:02.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:02 smithi183 ceph-mon[33745]: 161.2 deep-scrub ok 2024-03-26T04:21:02.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:02 smithi183 ceph-mon[33745]: pgmap v164: 484 pgs: 23 creating+peering, 96 unknown, 365 active+clean; 218 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 1.5 KiB/s rd, 59 KiB/s wr, 59 op/s 2024-03-26T04:21:02.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:02 smithi183 ceph-mon[33745]: from='client.48715 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74609-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:02.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:02 smithi183 ceph-mon[33745]: from='client.56174 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStat_smithi016-74264-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:02.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:02 smithi183 ceph-mon[33745]: osdmap e137: 8 total, 8 up, 8 in 2024-03-26T04:21:02.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:02 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:02.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:02 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-26T04:21:02.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:02 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi016-74514-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:02.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:02 smithi183 ceph-mon[33745]: from='client.24733 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi016-74514-1","app": "app1","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:03.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:03 smithi016 ceph-mon[37215]: from='client.24733 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi016-74514-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:03.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:03 smithi016 ceph-mon[37215]: osdmap e138: 8 total, 8 up, 8 in 2024-03-26T04:21:03.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:03 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2089077602' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi016-74285-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:03.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:03 smithi016 ceph-mon[37215]: from='client.56192 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi016-74285-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:03.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:03 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi016-74514-1","app": "app2"}]: dispatch 2024-03-26T04:21:03.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:03 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi016-74514-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:03.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:03 smithi016 ceph-mon[37215]: from='client.24733 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi016-74514-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:03.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:03 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:21:03.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:03 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:03.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:03 smithi016 ceph-mon[37215]: from='client.56192 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi016-74285-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:03.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:03 smithi016 ceph-mon[37215]: from='client.24733 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi016-74514-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:03.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:03 smithi016 ceph-mon[37215]: osdmap e139: 8 total, 8 up, 8 in 2024-03-26T04:21:03.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:03 smithi016 ceph-mon[29101]: from='client.24733 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi016-74514-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:03.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:03 smithi016 ceph-mon[29101]: osdmap e138: 8 total, 8 up, 8 in 2024-03-26T04:21:03.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:03 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2089077602' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi016-74285-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:03.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:03 smithi016 ceph-mon[29101]: from='client.56192 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi016-74285-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:03.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:03 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi016-74514-1","app": "app2"}]: dispatch 2024-03-26T04:21:03.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:03 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi016-74514-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:03.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:03 smithi016 ceph-mon[29101]: from='client.24733 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi016-74514-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:03.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:03 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:21:03.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:03 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:03.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:03 smithi016 ceph-mon[29101]: from='client.56192 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi016-74285-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:03.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:03 smithi016 ceph-mon[29101]: from='client.24733 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi016-74514-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:03.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:03 smithi016 ceph-mon[29101]: osdmap e139: 8 total, 8 up, 8 in 2024-03-26T04:21:03.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:03 smithi183 ceph-mon[33745]: from='client.24733 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi016-74514-1","app": "app1","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:03.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:03 smithi183 ceph-mon[33745]: osdmap e138: 8 total, 8 up, 8 in 2024-03-26T04:21:03.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:03 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2089077602' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi016-74285-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:03.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:03 smithi183 ceph-mon[33745]: from='client.56192 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi016-74285-16","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:03.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:03 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi016-74514-1","app": "app2"}]: dispatch 2024-03-26T04:21:03.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:03 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi016-74514-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:03.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:03 smithi183 ceph-mon[33745]: from='client.24733 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi016-74514-1","app": "app2","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:03.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:03 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:21:03.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:03 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:03.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:03 smithi183 ceph-mon[33745]: from='client.56192 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP_smithi016-74285-16","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:03.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:03 smithi183 ceph-mon[33745]: from='client.24733 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi016-74514-1","app": "app2","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:03.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:03 smithi183 ceph-mon[33745]: osdmap e139: 8 total, 8 up, 8 in 2024-03-26T04:21:04.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:04 smithi016 ceph-mon[37215]: pgmap v167: 452 pgs: 23 creating+peering, 41 unknown, 388 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 72 op/s 2024-03-26T04:21:04.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:04 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3475267265' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi016-74264-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:04.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:04 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi016-74514-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-03-26T04:21:04.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:04 smithi016 ceph-mon[37215]: from='client.48739 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi016-74264-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:04.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:04 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi016-74514-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T04:21:04.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:04 smithi016 ceph-mon[37215]: from='client.24733 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi016-74514-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T04:21:04.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:04 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:04.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:04 smithi016 ceph-mon[37215]: from='client.48739 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi016-74264-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:04.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:04 smithi016 ceph-mon[37215]: from='client.24733 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi016-74514-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-26T04:21:04.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:04 smithi016 ceph-mon[37215]: osdmap e140: 8 total, 8 up, 8 in 2024-03-26T04:21:04.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:04 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi016-74514-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T04:21:04.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:04 smithi016 ceph-mon[37215]: from='client.24733 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi016-74514-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T04:21:04.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:04 smithi016 ceph-mon[29101]: pgmap v167: 452 pgs: 23 creating+peering, 41 unknown, 388 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 72 op/s 2024-03-26T04:21:04.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:04 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3475267265' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi016-74264-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:04.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:04 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi016-74514-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-03-26T04:21:04.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:04 smithi016 ceph-mon[29101]: from='client.48739 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi016-74264-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:04.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:04 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi016-74514-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T04:21:04.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:04 smithi016 ceph-mon[29101]: from='client.24733 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi016-74514-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T04:21:04.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:04 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:04.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:04 smithi016 ceph-mon[29101]: from='client.48739 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi016-74264-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:04.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:04 smithi016 ceph-mon[29101]: from='client.24733 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi016-74514-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-26T04:21:04.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:04 smithi016 ceph-mon[29101]: osdmap e140: 8 total, 8 up, 8 in 2024-03-26T04:21:04.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:04 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi016-74514-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T04:21:04.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:04 smithi016 ceph-mon[29101]: from='client.24733 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi016-74514-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T04:21:04.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:04 smithi183 ceph-mon[33745]: pgmap v167: 452 pgs: 23 creating+peering, 41 unknown, 388 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 55 KiB/s rd, 72 op/s 2024-03-26T04:21:04.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:04 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3475267265' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi016-74264-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:04.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:04 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi016-74514-1","app":"dne","key":"key1","value":"value1"}]: dispatch 2024-03-26T04:21:04.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:04 smithi183 ceph-mon[33745]: from='client.48739 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi016-74264-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:04.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:04 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi016-74514-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T04:21:04.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:04 smithi183 ceph-mon[33745]: from='client.24733 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi016-74514-1","app":"app1","key":"key1","value":"value1"}]: dispatch 2024-03-26T04:21:04.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:04 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:04.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:04 smithi183 ceph-mon[33745]: from='client.48739 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi016-74264-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:04.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:04 smithi183 ceph-mon[33745]: from='client.24733 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi016-74514-1","app":"app1","key":"key1","value":"value1"}]': finished 2024-03-26T04:21:04.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:04 smithi183 ceph-mon[33745]: osdmap e140: 8 total, 8 up, 8 in 2024-03-26T04:21:04.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:04 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi016-74514-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T04:21:04.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:04 smithi183 ceph-mon[33745]: from='client.24733 ' entity='client.admin' cmd=[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi016-74514-1","app":"app1","key":"key2","value":"value2"}]: dispatch 2024-03-26T04:21:05.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:05 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/968806103' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi016-74609-16"}]: dispatch 2024-03-26T04:21:05.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:05 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:05.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:05 smithi016 ceph-mon[37215]: from='client.56231 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi016-74609-16"}]: dispatch 2024-03-26T04:21:05.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:05 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/968806103' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi016-74609-16"}]: dispatch 2024-03-26T04:21:05.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:05 smithi016 ceph-mon[37215]: from='client.56231 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi016-74609-16"}]: dispatch 2024-03-26T04:21:05.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:05 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/968806103' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi016-74609-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:05.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:05 smithi016 ceph-mon[37215]: from='client.56231 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi016-74609-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:05.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:05 smithi016 ceph-mon[37215]: from='client.24733 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi016-74514-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-26T04:21:05.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:05 smithi016 ceph-mon[37215]: from='client.56231 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi016-74609-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:21:05.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:05 smithi016 ceph-mon[37215]: osdmap e141: 8 total, 8 up, 8 in 2024-03-26T04:21:05.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:05 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/968806103' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi016-74609-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi016-74609-16"}]: dispatch 2024-03-26T04:21:05.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:05 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3781895997' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi016-74285-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:05.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:05 smithi016 ceph-mon[37215]: from='client.56231 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi016-74609-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi016-74609-16"}]: dispatch 2024-03-26T04:21:05.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:05 smithi016 ceph-mon[37215]: from='client.56225 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi016-74285-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:05.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:05 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi016-74514-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T04:21:05.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:05 smithi016 ceph-mon[37215]: from='client.24733 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi016-74514-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T04:21:05.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:05 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/968806103' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi016-74609-16"}]: dispatch 2024-03-26T04:21:05.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:05 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:05.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:05 smithi016 ceph-mon[29101]: from='client.56231 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi016-74609-16"}]: dispatch 2024-03-26T04:21:05.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:05 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/968806103' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi016-74609-16"}]: dispatch 2024-03-26T04:21:05.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:05 smithi016 ceph-mon[29101]: from='client.56231 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi016-74609-16"}]: dispatch 2024-03-26T04:21:05.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:05 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/968806103' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi016-74609-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:05.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:05 smithi016 ceph-mon[29101]: from='client.56231 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi016-74609-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:05.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:05 smithi016 ceph-mon[29101]: from='client.24733 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi016-74514-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-26T04:21:05.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:05 smithi016 ceph-mon[29101]: from='client.56231 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi016-74609-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:21:05.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:05 smithi016 ceph-mon[29101]: osdmap e141: 8 total, 8 up, 8 in 2024-03-26T04:21:05.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:05 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/968806103' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi016-74609-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi016-74609-16"}]: dispatch 2024-03-26T04:21:05.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:05 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3781895997' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi016-74285-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:05.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:05 smithi016 ceph-mon[29101]: from='client.56231 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi016-74609-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi016-74609-16"}]: dispatch 2024-03-26T04:21:05.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:05 smithi016 ceph-mon[29101]: from='client.56225 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi016-74285-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:05.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:05 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi016-74514-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T04:21:05.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:05 smithi016 ceph-mon[29101]: from='client.24733 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi016-74514-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T04:21:05.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:05 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/968806103' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi016-74609-16"}]: dispatch 2024-03-26T04:21:05.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:05 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:05.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:05 smithi183 ceph-mon[33745]: from='client.56231 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi016-74609-16"}]: dispatch 2024-03-26T04:21:05.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:05 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/968806103' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi016-74609-16"}]: dispatch 2024-03-26T04:21:05.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:05 smithi183 ceph-mon[33745]: from='client.56231 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi016-74609-16"}]: dispatch 2024-03-26T04:21:05.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:05 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/968806103' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi016-74609-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:05.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:05 smithi183 ceph-mon[33745]: from='client.56231 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi016-74609-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:05.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:05 smithi183 ceph-mon[33745]: from='client.24733 ' entity='client.admin' cmd='[{"prefix":"osd pool application set","pool":"LibRadosMiscPP_smithi016-74514-1","app":"app1","key":"key2","value":"value2"}]': finished 2024-03-26T04:21:05.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:05 smithi183 ceph-mon[33745]: from='client.56231 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsECPP_smithi016-74609-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:21:05.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:05 smithi183 ceph-mon[33745]: osdmap e141: 8 total, 8 up, 8 in 2024-03-26T04:21:05.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:05 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/968806103' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi016-74609-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi016-74609-16"}]: dispatch 2024-03-26T04:21:05.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:05 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3781895997' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi016-74285-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:05.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:05 smithi183 ceph-mon[33745]: from='client.56231 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi016-74609-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi016-74609-16"}]: dispatch 2024-03-26T04:21:05.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:05 smithi183 ceph-mon[33745]: from='client.56225 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi016-74285-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:05.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:05 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2835389274' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi016-74514-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T04:21:05.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:05 smithi183 ceph-mon[33745]: from='client.24733 ' entity='client.admin' cmd=[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi016-74514-1","app":"app1","key":"key1"}]: dispatch 2024-03-26T04:21:06.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:06 smithi016 ceph-mon[29101]: pgmap v170: 420 pgs: 13 creating+peering, 51 unknown, 356 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 75 op/s 2024-03-26T04:21:06.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:06 smithi016 ceph-mon[29101]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:21:06.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:06 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:06.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:06 smithi016 ceph-mon[29101]: from='client.56225 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi016-74285-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:06.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:06 smithi016 ceph-mon[37215]: pgmap v170: 420 pgs: 13 creating+peering, 51 unknown, 356 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 75 op/s 2024-03-26T04:21:06.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:06 smithi016 ceph-mon[37215]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:21:06.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:06 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:06.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:06 smithi016 ceph-mon[37215]: from='client.56225 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi016-74285-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:06.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:06 smithi183 ceph-mon[33745]: pgmap v170: 420 pgs: 13 creating+peering, 51 unknown, 356 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 56 KiB/s rd, 0 B/s wr, 75 op/s 2024-03-26T04:21:06.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:06 smithi183 ceph-mon[33745]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:21:06.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:06 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:06.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:06 smithi183 ceph-mon[33745]: from='client.56225 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi016-74285-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:07.108 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-03-26T04:21:07.108 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-03-26T04:21:07.108 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-03-26T04:21:07.108 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-03-26T04:21:07.108 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-03-26T04:21:07.108 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-03-26T04:21:07.108 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: 2024-03-26T04:21:07.108 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-03-26T04:21:07.109 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: seed 74514 2024-03-26T04:21:07.109 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-03-26T04:21:07.109 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (2 ms) 2024-03-26T04:21:07.109 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-03-26T04:21:07.109 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (371 ms) 2024-03-26T04:21:07.109 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-03-26T04:21:07.109 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (10 ms) 2024-03-26T04:21:07.109 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-03-26T04:21:07.109 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (7 ms) 2024-03-26T04:21:07.109 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-03-26T04:21:07.109 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (7 ms) 2024-03-26T04:21:07.109 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-03-26T04:21:07.109 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (3 ms) 2024-03-26T04:21:07.109 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-03-26T04:21:07.110 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (3 ms) 2024-03-26T04:21:07.110 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-03-26T04:21:07.110 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (7 ms) 2024-03-26T04:21:07.110 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-03-26T04:21:07.110 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (3 ms) 2024-03-26T04:21:07.110 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-03-26T04:21:07.110 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (15 ms) 2024-03-26T04:21:07.110 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-03-26T04:21:07.110 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (5 ms) 2024-03-26T04:21:07.110 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-03-26T04:21:07.110 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (7 ms) 2024-03-26T04:21:07.110 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-03-26T04:21:07.110 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (11 ms) 2024-03-26T04:21:07.111 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-03-26T04:21:07.111 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: osd_max_attr_size = 0 2024-03-26T04:21:07.111 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-03-26T04:21:07.111 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (1485 ms) 2024-03-26T04:21:07.111 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-03-26T04:21:07.111 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (343 ms) 2024-03-26T04:21:07.111 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-03-26T04:21:07.111 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-03-26T04:21:07.111 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: done waiting, doing copies 2024-03-26T04:21:07.111 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: waiting for final deep scrubs... 2024-03-26T04:21:07.111 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: done waiting 2024-03-26T04:21:07.111 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (60869 ms) 2024-03-26T04:21:07.111 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-03-26T04:21:07.111 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (3 ms) 2024-03-26T04:21:07.112 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-03-26T04:21:07.112 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (5 ms) 2024-03-26T04:21:07.112 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-03-26T04:21:07.112 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (4698 ms) 2024-03-26T04:21:07.112 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-03-26T04:21:07.112 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-03-26T04:21:07.112 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-03-26T04:21:07.112 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-03-26T04:21:07.112 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-03-26T04:21:07.112 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-03-26T04:21:07.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:07 smithi016 ceph-mon[37215]: from='client.24733 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi016-74514-1","app":"app1","key":"key1"}]': finished 2024-03-26T04:21:07.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:07 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:07.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:07 smithi016 ceph-mon[37215]: osdmap e142: 8 total, 8 up, 8 in 2024-03-26T04:21:07.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:07 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3678621059' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi016-74264-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:07.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:07 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:21:07.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:07 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:21:07.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:07 smithi016 ceph-mon[37215]: from='client.56231 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi016-74609-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi016-74609-16"}]': finished 2024-03-26T04:21:07.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:07 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3678621059' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi016-74264-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:07.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:07 smithi016 ceph-mon[37215]: osdmap e143: 8 total, 8 up, 8 in 2024-03-26T04:21:07.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:07 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/4124277665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi016-74514-24"}]: dispatch 2024-03-26T04:21:07.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:07 smithi016 ceph-mon[29101]: from='client.24733 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi016-74514-1","app":"app1","key":"key1"}]': finished 2024-03-26T04:21:07.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:07 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:07.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:07 smithi016 ceph-mon[29101]: osdmap e142: 8 total, 8 up, 8 in 2024-03-26T04:21:07.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:07 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3678621059' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi016-74264-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:07.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:07 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:21:07.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:07 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:21:07.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:07 smithi016 ceph-mon[29101]: from='client.56231 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi016-74609-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi016-74609-16"}]': finished 2024-03-26T04:21:07.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:07 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3678621059' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi016-74264-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:07.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:07 smithi016 ceph-mon[29101]: osdmap e143: 8 total, 8 up, 8 in 2024-03-26T04:21:07.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:07 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/4124277665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi016-74514-24"}]: dispatch 2024-03-26T04:21:07.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:07 smithi183 ceph-mon[33745]: from='client.24733 ' entity='client.admin' cmd='[{"prefix":"osd pool application rm","pool":"LibRadosMiscPP_smithi016-74514-1","app":"app1","key":"key1"}]': finished 2024-03-26T04:21:07.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:07 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:07.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:07 smithi183 ceph-mon[33745]: osdmap e142: 8 total, 8 up, 8 in 2024-03-26T04:21:07.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:07 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3678621059' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi016-74264-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:07.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:07 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:21:07.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:07 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:21:07.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:07 smithi183 ceph-mon[33745]: from='client.56231 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsECPP_smithi016-74609-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsECPP_smithi016-74609-16"}]': finished 2024-03-26T04:21:07.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:07 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3678621059' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "Operate2Mtime_smithi016-74264-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:07.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:07 smithi183 ceph-mon[33745]: osdmap e143: 8 total, 8 up, 8 in 2024-03-26T04:21:07.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:07 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/4124277665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi016-74514-24"}]: dispatch 2024-03-26T04:21:08.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:08 smithi016 ceph-mon[29101]: pgmap v173: 452 pgs: 4 creating+peering, 86 unknown, 362 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 255 B/s wr, 0 op/s 2024-03-26T04:21:08.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:08 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:08.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:08 smithi016 ceph-mon[29101]: from='client.56243 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi016-74514-24"}]: dispatch 2024-03-26T04:21:08.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:08 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/4124277665' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi016-74514-24"}]: dispatch 2024-03-26T04:21:08.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:08 smithi016 ceph-mon[29101]: from='client.56243 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi016-74514-24"}]: dispatch 2024-03-26T04:21:08.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:08 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/4124277665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi016-74514-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:08.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:08 smithi016 ceph-mon[29101]: from='client.56243 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi016-74514-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:08.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:08 smithi016 ceph-mon[29101]: from='client.56243 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi016-74514-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:21:08.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:08 smithi016 ceph-mon[29101]: osdmap e144: 8 total, 8 up, 8 in 2024-03-26T04:21:08.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:08 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/4124277665' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi016-74514-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi016-74514-24"}]: dispatch 2024-03-26T04:21:08.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:08 smithi016 ceph-mon[29101]: from='client.56243 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi016-74514-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi016-74514-24"}]: dispatch 2024-03-26T04:21:08.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:08 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1402460973' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi016-74285-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:08.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:08 smithi016 ceph-mon[29101]: from='client.56240 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi016-74285-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:08.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:08 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:08.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:08 smithi016 ceph-mon[37215]: pgmap v173: 452 pgs: 4 creating+peering, 86 unknown, 362 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 255 B/s wr, 0 op/s 2024-03-26T04:21:08.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:08 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:08.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:08 smithi016 ceph-mon[37215]: from='client.56243 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi016-74514-24"}]: dispatch 2024-03-26T04:21:08.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:08 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/4124277665' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi016-74514-24"}]: dispatch 2024-03-26T04:21:08.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:08 smithi016 ceph-mon[37215]: from='client.56243 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi016-74514-24"}]: dispatch 2024-03-26T04:21:08.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:08 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/4124277665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi016-74514-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:08.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:08 smithi016 ceph-mon[37215]: from='client.56243 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi016-74514-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:08.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:08 smithi016 ceph-mon[37215]: from='client.56243 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi016-74514-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:21:08.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:08 smithi016 ceph-mon[37215]: osdmap e144: 8 total, 8 up, 8 in 2024-03-26T04:21:08.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:08 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/4124277665' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi016-74514-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi016-74514-24"}]: dispatch 2024-03-26T04:21:08.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:08 smithi016 ceph-mon[37215]: from='client.56243 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi016-74514-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi016-74514-24"}]: dispatch 2024-03-26T04:21:08.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:08 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1402460973' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi016-74285-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:08.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:08 smithi016 ceph-mon[37215]: from='client.56240 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi016-74285-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:08.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:08 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:08.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:08 smithi183 ceph-mon[33745]: pgmap v173: 452 pgs: 4 creating+peering, 86 unknown, 362 active+clean; 217 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 255 B/s rd, 255 B/s wr, 0 op/s 2024-03-26T04:21:08.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:08 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:08.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:08 smithi183 ceph-mon[33745]: from='client.56243 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi016-74514-24"}]: dispatch 2024-03-26T04:21:08.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:08 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/4124277665' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi016-74514-24"}]: dispatch 2024-03-26T04:21:08.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:08 smithi183 ceph-mon[33745]: from='client.56243 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi016-74514-24"}]: dispatch 2024-03-26T04:21:08.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:08 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/4124277665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi016-74514-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:08.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:08 smithi183 ceph-mon[33745]: from='client.56243 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi016-74514-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:08.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:08 smithi183 ceph-mon[33745]: from='client.56243 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi016-74514-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:21:08.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:08 smithi183 ceph-mon[33745]: osdmap e144: 8 total, 8 up, 8 in 2024-03-26T04:21:08.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:08 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/4124277665' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi016-74514-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi016-74514-24"}]: dispatch 2024-03-26T04:21:08.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:08 smithi183 ceph-mon[33745]: from='client.56243 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi016-74514-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi016-74514-24"}]: dispatch 2024-03-26T04:21:08.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:08 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1402460973' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi016-74285-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:08.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:08 smithi183 ceph-mon[33745]: from='client.56240 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi016-74285-18","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:08.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:08 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:09.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:21:08 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:21:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:21:10.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:10 smithi016 ceph-mon[37215]: pgmap v176: 396 pgs: 1 creating+activating, 21 creating+peering, 38 unknown, 336 active+clean; 592 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-26T04:21:10.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:10 smithi016 ceph-mon[37215]: from='client.56240 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi016-74285-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:10.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:10 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:10.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:10 smithi016 ceph-mon[37215]: osdmap e145: 8 total, 8 up, 8 in 2024-03-26T04:21:10.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:10 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3177131169' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi016-74264-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:10.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:10 smithi016 ceph-mon[37215]: from='client.56249 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi016-74264-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:10.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:10 smithi016 ceph-mon[29101]: pgmap v176: 396 pgs: 1 creating+activating, 21 creating+peering, 38 unknown, 336 active+clean; 592 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-26T04:21:10.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:10 smithi016 ceph-mon[29101]: from='client.56240 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi016-74285-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:10.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:10 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:10.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:10 smithi016 ceph-mon[29101]: osdmap e145: 8 total, 8 up, 8 in 2024-03-26T04:21:10.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:10 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3177131169' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi016-74264-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:10.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:10 smithi016 ceph-mon[29101]: from='client.56249 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi016-74264-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:10.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:10 smithi183 ceph-mon[33745]: pgmap v176: 396 pgs: 1 creating+activating, 21 creating+peering, 38 unknown, 336 active+clean; 592 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-26T04:21:10.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:10 smithi183 ceph-mon[33745]: from='client.56240 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP_smithi016-74285-18","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:10.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:10 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:10.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:10 smithi183 ceph-mon[33745]: osdmap e145: 8 total, 8 up, 8 in 2024-03-26T04:21:10.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:10 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3177131169' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi016-74264-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:10.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:10 smithi183 ceph-mon[33745]: from='client.56249 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi016-74264-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:11.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:11 smithi016 ceph-mon[37215]: from='client.56243 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi016-74514-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi016-74514-24"}]': finished 2024-03-26T04:21:11.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:11 smithi016 ceph-mon[37215]: from='client.56249 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi016-74264-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:11.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:11 smithi016 ceph-mon[37215]: osdmap e146: 8 total, 8 up, 8 in 2024-03-26T04:21:11.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:11 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:11.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:11 smithi016 ceph-mon[37215]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:21:11.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:11 smithi016 ceph-mon[29101]: from='client.56243 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi016-74514-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi016-74514-24"}]': finished 2024-03-26T04:21:11.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:11 smithi016 ceph-mon[29101]: from='client.56249 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi016-74264-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:11.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:11 smithi016 ceph-mon[29101]: osdmap e146: 8 total, 8 up, 8 in 2024-03-26T04:21:11.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:11 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:11.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:11 smithi016 ceph-mon[29101]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:21:11.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:11 smithi183 ceph-mon[33745]: from='client.56243 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi016-74514-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi016-74514-24"}]': finished 2024-03-26T04:21:11.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:11 smithi183 ceph-mon[33745]: from='client.56249 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatNS_smithi016-74264-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:11.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:11 smithi183 ceph-mon[33745]: osdmap e146: 8 total, 8 up, 8 in 2024-03-26T04:21:11.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:11 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:11.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:11 smithi183 ceph-mon[33745]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:21:12.111 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP api_snapshots_pp: Running main() from gmock_main.cc 2024-03-26T04:21:12.111 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-03-26T04:21:12.111 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-03-26T04:21:12.111 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-03-26T04:21:12.111 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: seed 74609 2024-03-26T04:21:12.111 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-03-26T04:21:12.111 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (2381 ms) 2024-03-26T04:21:12.111 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-03-26T04:21:12.111 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (2186 ms) 2024-03-26T04:21:12.111 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-03-26T04:21:12.111 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (2110 ms) 2024-03-26T04:21:12.111 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-03-26T04:21:12.112 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (2064 ms) 2024-03-26T04:21:12.112 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-03-26T04:21:12.112 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (5910 ms) 2024-03-26T04:21:12.112 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (14651 ms total) 2024-03-26T04:21:12.112 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: 2024-03-26T04:21:12.112 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-03-26T04:21:12.112 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-03-26T04:21:12.112 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (4594 ms) 2024-03-26T04:21:12.112 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-03-26T04:21:12.112 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (4735 ms) 2024-03-26T04:21:12.112 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-03-26T04:21:12.112 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (7278 ms) 2024-03-26T04:21:12.112 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-03-26T04:21:12.112 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (3942 ms) 2024-03-26T04:21:12.113 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-03-26T04:21:12.113 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (2084 ms) 2024-03-26T04:21:12.113 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-03-26T04:21:12.113 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-1775-g5e70124b/rpm/el9/BUILD/ceph-19.0.0-1775-g5e70124b/src/test/librados/snapshots_cxx.cc:463: Skipped 2024-03-26T04:21:12.113 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: 2024-03-26T04:21:12.113 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-03-26T04:21:12.113 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-03-26T04:21:12.113 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: deleting snap 14 in pool LibRadosSnapshotsSelfManagedPP_smithi016-74609-7 2024-03-26T04:21:12.113 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: waiting for snaps to purge 2024-03-26T04:21:12.113 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (17921 ms) 2024-03-26T04:21:12.113 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (40555 ms total) 2024-03-26T04:21:12.113 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: 2024-03-26T04:21:12.113 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-03-26T04:21:12.113 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-03-26T04:21:12.113 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (9 ms) 2024-03-26T04:21:12.114 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-03-26T04:21:12.114 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (5052 ms) 2024-03-26T04:21:12.114 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (5061 ms total) 2024-03-26T04:21:12.114 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: 2024-03-26T04:21:12.114 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-03-26T04:21:12.114 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-03-26T04:21:12.114 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (3014 ms) 2024-03-26T04:21:12.114 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-03-26T04:21:12.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:12 smithi016 ceph-mon[37215]: pgmap v179: 404 pgs: 1 creating+activating, 21 creating+peering, 46 unknown, 336 active+clean; 592 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-26T04:21:12.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:12 smithi016 ceph-mon[37215]: osdmap e147: 8 total, 8 up, 8 in 2024-03-26T04:21:12.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:12 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/4124277665' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74514-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:12.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:12 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2291501455' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi016-74285-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:12.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:12 smithi016 ceph-mon[37215]: from='client.56243 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74514-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:12.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:12 smithi016 ceph-mon[37215]: from='client.48784 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi016-74285-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:12.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:12 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:12.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:12 smithi016 ceph-mon[29101]: pgmap v179: 404 pgs: 1 creating+activating, 21 creating+peering, 46 unknown, 336 active+clean; 592 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-26T04:21:12.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:12 smithi016 ceph-mon[29101]: osdmap e147: 8 total, 8 up, 8 in 2024-03-26T04:21:12.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:12 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/4124277665' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74514-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:12.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:12 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2291501455' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi016-74285-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:12.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:12 smithi016 ceph-mon[29101]: from='client.56243 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74514-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:12.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:12 smithi016 ceph-mon[29101]: from='client.48784 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi016-74285-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:12.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:12 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:12.542 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:21:12 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:21:12.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:12 smithi183 ceph-mon[33745]: pgmap v179: 404 pgs: 1 creating+activating, 21 creating+peering, 46 unknown, 336 active+clean; 592 KiB data, 1.1 GiB used, 714 GiB / 715 GiB avail 2024-03-26T04:21:12.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:12 smithi183 ceph-mon[33745]: osdmap e147: 8 total, 8 up, 8 in 2024-03-26T04:21:12.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:12 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/4124277665' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74514-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:12.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:12 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2291501455' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi016-74285-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:12.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:12 smithi183 ceph-mon[33745]: from='client.56243 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74514-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:12.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:12 smithi183 ceph-mon[33745]: from='client.48784 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi016-74285-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:12.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:12 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:13.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:13 smithi016 ceph-mon[37215]: from='client.56243 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74514-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:13.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:13 smithi016 ceph-mon[37215]: from='client.48784 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi016-74285-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:13.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:13 smithi016 ceph-mon[37215]: osdmap e148: 8 total, 8 up, 8 in 2024-03-26T04:21:13.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:13 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/4124277665' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74514-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:13.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:13 smithi016 ceph-mon[37215]: from='client.56243 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74514-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:13.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:13 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3135545303' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi016-74264-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:13.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:13 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:21:13.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:13 smithi016 ceph-mon[37215]: from='client.48790 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi016-74264-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:13.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:13 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:13.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:13 smithi016 ceph-mon[37215]: from='client.56243 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74514-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:13.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:13 smithi016 ceph-mon[29101]: from='client.56243 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74514-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:13.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:13 smithi016 ceph-mon[29101]: from='client.48784 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi016-74285-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:13.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:13 smithi016 ceph-mon[29101]: osdmap e148: 8 total, 8 up, 8 in 2024-03-26T04:21:13.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:13 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/4124277665' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74514-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:13.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:13 smithi016 ceph-mon[29101]: from='client.56243 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74514-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:13.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:13 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3135545303' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi016-74264-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:13.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:13 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:21:13.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:13 smithi016 ceph-mon[29101]: from='client.48790 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi016-74264-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:13.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:13 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:13.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:13 smithi016 ceph-mon[29101]: from='client.56243 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74514-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:13.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:13 smithi183 ceph-mon[33745]: from='client.56243 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74514-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:13.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:13 smithi183 ceph-mon[33745]: from='client.48784 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteSamePP2_smithi016-74285-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:13.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:13 smithi183 ceph-mon[33745]: osdmap e148: 8 total, 8 up, 8 in 2024-03-26T04:21:13.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:13 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/4124277665' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74514-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:13.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:13 smithi183 ceph-mon[33745]: from='client.56243 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74514-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:13.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:13 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3135545303' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi016-74264-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:13.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:13 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:21:13.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:13 smithi183 ceph-mon[33745]: from='client.48790 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemove_smithi016-74264-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:13.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:13 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:13.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:13 smithi183 ceph-mon[33745]: from='client.56243 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74514-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:14.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:14 smithi016 ceph-mon[37215]: pgmap v182: 468 pgs: 50 creating+peering, 22 unknown, 396 active+clean; 592 KiB data, 550 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:21:14.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:14 smithi016 ceph-mon[37215]: from='client.48790 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi016-74264-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:14.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:14 smithi016 ceph-mon[37215]: osdmap e149: 8 total, 8 up, 8 in 2024-03-26T04:21:14.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:14 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:14.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:14 smithi016 ceph-mon[29101]: pgmap v182: 468 pgs: 50 creating+peering, 22 unknown, 396 active+clean; 592 KiB data, 550 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:21:14.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:14 smithi016 ceph-mon[29101]: from='client.48790 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi016-74264-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:14.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:14 smithi016 ceph-mon[29101]: osdmap e149: 8 total, 8 up, 8 in 2024-03-26T04:21:14.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:14 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:14.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:14 smithi183 ceph-mon[33745]: pgmap v182: 468 pgs: 50 creating+peering, 22 unknown, 396 active+clean; 592 KiB data, 550 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:21:14.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:14 smithi183 ceph-mon[33745]: from='client.48790 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemove_smithi016-74264-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:14.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:14 smithi183 ceph-mon[33745]: osdmap e149: 8 total, 8 up, 8 in 2024-03-26T04:21:14.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:14 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:15.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:15 smithi183 ceph-mon[33745]: osdmap e150: 8 total, 8 up, 8 in 2024-03-26T04:21:15.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:15 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1502944146' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi016-74285-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:15.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:15 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:15.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:15 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1502944146' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi016-74285-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:15.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:15 smithi183 ceph-mon[33745]: osdmap e151: 8 total, 8 up, 8 in 2024-03-26T04:21:15.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:15 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/4124277665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi016-74514-24"}]: dispatch 2024-03-26T04:21:15.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:15 smithi183 ceph-mon[33745]: from='client.56243 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi016-74514-24"}]: dispatch 2024-03-26T04:21:15.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:15 smithi016 ceph-mon[37215]: osdmap e150: 8 total, 8 up, 8 in 2024-03-26T04:21:15.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:15 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1502944146' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi016-74285-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:15.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:15 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:15.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:15 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1502944146' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi016-74285-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:15.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:15 smithi016 ceph-mon[37215]: osdmap e151: 8 total, 8 up, 8 in 2024-03-26T04:21:15.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:15 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/4124277665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi016-74514-24"}]: dispatch 2024-03-26T04:21:15.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:15 smithi016 ceph-mon[37215]: from='client.56243 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi016-74514-24"}]: dispatch 2024-03-26T04:21:15.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:15 smithi016 ceph-mon[29101]: osdmap e150: 8 total, 8 up, 8 in 2024-03-26T04:21:15.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:15 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1502944146' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi016-74285-20","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:15.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:15 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:15.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:15 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1502944146' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPPNS_smithi016-74285-20","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:15.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:15 smithi016 ceph-mon[29101]: osdmap e151: 8 total, 8 up, 8 in 2024-03-26T04:21:15.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:15 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/4124277665' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi016-74514-24"}]: dispatch 2024-03-26T04:21:15.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:15 smithi016 ceph-mon[29101]: from='client.56243 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi016-74514-24"}]: dispatch 2024-03-26T04:21:16.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:16 smithi016 ceph-mon[37215]: pgmap v185: 404 pgs: 32 unknown, 372 active+clean; 592 KiB data, 550 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:21:16.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:16 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1784976523' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi016-74264-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:16.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:16 smithi016 ceph-mon[37215]: from='client.48799 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi016-74264-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:16.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:16 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:16.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:16 smithi016 ceph-mon[37215]: from='client.56243 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi016-74514-24"}]': finished 2024-03-26T04:21:16.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:16 smithi016 ceph-mon[37215]: from='client.48799 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi016-74264-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:16.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:16 smithi016 ceph-mon[37215]: osdmap e152: 8 total, 8 up, 8 in 2024-03-26T04:21:16.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:16 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/4124277665' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi016-74514-24"}]: dispatch 2024-03-26T04:21:16.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:16 smithi016 ceph-mon[37215]: from='client.56243 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi016-74514-24"}]: dispatch 2024-03-26T04:21:16.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:16 smithi016 ceph-mon[29101]: pgmap v185: 404 pgs: 32 unknown, 372 active+clean; 592 KiB data, 550 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:21:16.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:16 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1784976523' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi016-74264-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:16.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:16 smithi016 ceph-mon[29101]: from='client.48799 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi016-74264-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:16.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:16 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:16.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:16 smithi016 ceph-mon[29101]: from='client.56243 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi016-74514-24"}]': finished 2024-03-26T04:21:16.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:16 smithi016 ceph-mon[29101]: from='client.48799 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi016-74264-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:16.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:16 smithi016 ceph-mon[29101]: osdmap e152: 8 total, 8 up, 8 in 2024-03-26T04:21:16.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:16 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/4124277665' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi016-74514-24"}]: dispatch 2024-03-26T04:21:16.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:16 smithi016 ceph-mon[29101]: from='client.56243 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi016-74514-24"}]: dispatch 2024-03-26T04:21:16.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:16 smithi183 ceph-mon[33745]: pgmap v185: 404 pgs: 32 unknown, 372 active+clean; 592 KiB data, 550 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:21:16.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:16 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1784976523' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi016-74264-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:16.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:16 smithi183 ceph-mon[33745]: from='client.48799 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi016-74264-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:16.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:16 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:16.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:16 smithi183 ceph-mon[33745]: from='client.56243 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi016-74514-24"}]': finished 2024-03-26T04:21:16.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:16 smithi183 ceph-mon[33745]: from='client.48799 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClass_smithi016-74264-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:16.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:16 smithi183 ceph-mon[33745]: osdmap e152: 8 total, 8 up, 8 in 2024-03-26T04:21:16.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:16 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/4124277665' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi016-74514-24"}]: dispatch 2024-03-26T04:21:16.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:16 smithi183 ceph-mon[33745]: from='client.56243 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi016-74514-24"}]: dispatch 2024-03-26T04:21:17.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:17 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:17.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:17 smithi016 ceph-mon[29101]: from='client.56243 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi016-74514-24"}]': finished 2024-03-26T04:21:17.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:17 smithi016 ceph-mon[29101]: osdmap e153: 8 total, 8 up, 8 in 2024-03-26T04:21:17.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:17 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/968806103' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi016-74609-16"}]: dispatch 2024-03-26T04:21:17.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:17 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/71736637' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi016-74285-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:17.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:17 smithi016 ceph-mon[29101]: from='client.56231 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi016-74609-16"}]: dispatch 2024-03-26T04:21:17.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:17 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:17.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:17 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:17.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:17 smithi016 ceph-mon[37215]: from='client.56243 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi016-74514-24"}]': finished 2024-03-26T04:21:17.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:17 smithi016 ceph-mon[37215]: osdmap e153: 8 total, 8 up, 8 in 2024-03-26T04:21:17.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:17 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/968806103' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi016-74609-16"}]: dispatch 2024-03-26T04:21:17.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:17 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/71736637' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi016-74285-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:17.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:17 smithi016 ceph-mon[37215]: from='client.56231 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi016-74609-16"}]: dispatch 2024-03-26T04:21:17.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:17 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:18.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:17 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:18.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:17 smithi183 ceph-mon[33745]: from='client.56243 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi016-74514-24"}]': finished 2024-03-26T04:21:18.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:17 smithi183 ceph-mon[33745]: osdmap e153: 8 total, 8 up, 8 in 2024-03-26T04:21:18.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:17 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/968806103' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi016-74609-16"}]: dispatch 2024-03-26T04:21:18.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:17 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/71736637' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi016-74285-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:18.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:17 smithi183 ceph-mon[33745]: from='client.56231 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi016-74609-16"}]: dispatch 2024-03-26T04:21:18.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:17 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:18.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:21:18 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:21:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:21:18.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:18 smithi016 ceph-mon[37215]: pgmap v188: 396 pgs: 27 unknown, 369 active+clean; 592 KiB data, 554 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:21:18.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:18 smithi016 ceph-mon[37215]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:21:18.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:18 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/71736637' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi016-74285-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:18.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:18 smithi016 ceph-mon[37215]: from='client.56231 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi016-74609-16"}]': finished 2024-03-26T04:21:18.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:18 smithi016 ceph-mon[37215]: osdmap e154: 8 total, 8 up, 8 in 2024-03-26T04:21:18.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:18 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/968806103' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi016-74609-16"}]: dispatch 2024-03-26T04:21:18.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:18 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2202019513' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi016-74264-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:18.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:18 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3347595959' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi016-74514-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:18.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:18 smithi016 ceph-mon[37215]: from='client.56231 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi016-74609-16"}]: dispatch 2024-03-26T04:21:18.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:18 smithi016 ceph-mon[37215]: from='client.48814 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi016-74264-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:18.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:18 smithi016 ceph-mon[37215]: from='client.48817 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi016-74514-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:18.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:18 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:18.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:18 smithi016 ceph-mon[29101]: pgmap v188: 396 pgs: 27 unknown, 369 active+clean; 592 KiB data, 554 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:21:18.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:18 smithi016 ceph-mon[29101]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:21:18.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:18 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/71736637' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi016-74285-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:18.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:18 smithi016 ceph-mon[29101]: from='client.56231 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi016-74609-16"}]': finished 2024-03-26T04:21:18.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:18 smithi016 ceph-mon[29101]: osdmap e154: 8 total, 8 up, 8 in 2024-03-26T04:21:18.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:18 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/968806103' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi016-74609-16"}]: dispatch 2024-03-26T04:21:18.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:18 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2202019513' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi016-74264-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:18.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:18 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3347595959' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi016-74514-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:18.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:18 smithi016 ceph-mon[29101]: from='client.56231 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi016-74609-16"}]: dispatch 2024-03-26T04:21:18.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:18 smithi016 ceph-mon[29101]: from='client.48814 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi016-74264-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:18.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:18 smithi016 ceph-mon[29101]: from='client.48817 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi016-74514-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:18.917 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:18 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:19.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:18 smithi183 ceph-mon[33745]: pgmap v188: 396 pgs: 27 unknown, 369 active+clean; 592 KiB data, 554 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:21:19.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:18 smithi183 ceph-mon[33745]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:21:19.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:18 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/71736637' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleStatPP_smithi016-74285-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:19.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:18 smithi183 ceph-mon[33745]: from='client.56231 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsECPP_smithi016-74609-16"}]': finished 2024-03-26T04:21:19.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:18 smithi183 ceph-mon[33745]: osdmap e154: 8 total, 8 up, 8 in 2024-03-26T04:21:19.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:18 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/968806103' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi016-74609-16"}]: dispatch 2024-03-26T04:21:19.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:18 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2202019513' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi016-74264-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:19.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:18 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3347595959' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi016-74514-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:19.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:18 smithi183 ceph-mon[33745]: from='client.56231 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi016-74609-16"}]: dispatch 2024-03-26T04:21:19.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:18 smithi183 ceph-mon[33745]: from='client.48814 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi016-74264-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:19.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:18 smithi183 ceph-mon[33745]: from='client.48817 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi016-74514-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:19.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:18 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:20.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:20 smithi016 ceph-mon[29101]: pgmap v191: 452 pgs: 14 creating+peering, 82 unknown, 356 active+clean; 592 KiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:21:20.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:20 smithi016 ceph-mon[29101]: from='client.56231 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi016-74609-16"}]': finished 2024-03-26T04:21:20.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:20 smithi016 ceph-mon[29101]: from='client.48814 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi016-74264-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:20.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:20 smithi016 ceph-mon[29101]: from='client.48817 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi016-74514-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:20.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:20 smithi016 ceph-mon[29101]: osdmap e155: 8 total, 8 up, 8 in 2024-03-26T04:21:20.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:20 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3873220630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21"}]: dispatch 2024-03-26T04:21:20.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:20 smithi016 ceph-mon[29101]: from='client.48826 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21"}]: dispatch 2024-03-26T04:21:20.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:20 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3873220630' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21"}]: dispatch 2024-03-26T04:21:20.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:20 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:20.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:20 smithi016 ceph-mon[29101]: from='client.48826 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21"}]: dispatch 2024-03-26T04:21:20.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:20 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3873220630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:20.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:20 smithi016 ceph-mon[29101]: from='client.48826 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:20.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:20 smithi016 ceph-mon[37215]: pgmap v191: 452 pgs: 14 creating+peering, 82 unknown, 356 active+clean; 592 KiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:21:20.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:20 smithi016 ceph-mon[37215]: from='client.56231 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi016-74609-16"}]': finished 2024-03-26T04:21:20.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:20 smithi016 ceph-mon[37215]: from='client.48814 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi016-74264-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:20.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:20 smithi016 ceph-mon[37215]: from='client.48817 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi016-74514-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:20.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:20 smithi016 ceph-mon[37215]: osdmap e155: 8 total, 8 up, 8 in 2024-03-26T04:21:20.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:20 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3873220630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21"}]: dispatch 2024-03-26T04:21:20.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:20 smithi016 ceph-mon[37215]: from='client.48826 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21"}]: dispatch 2024-03-26T04:21:20.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:20 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3873220630' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21"}]: dispatch 2024-03-26T04:21:20.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:20 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:20.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:20 smithi016 ceph-mon[37215]: from='client.48826 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21"}]: dispatch 2024-03-26T04:21:20.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:20 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3873220630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:20.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:20 smithi016 ceph-mon[37215]: from='client.48826 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:20.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:20 smithi183 ceph-mon[33745]: pgmap v191: 452 pgs: 14 creating+peering, 82 unknown, 356 active+clean; 592 KiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:21:20.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:20 smithi183 ceph-mon[33745]: from='client.56231 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsECPP_smithi016-74609-16"}]': finished 2024-03-26T04:21:20.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:20 smithi183 ceph-mon[33745]: from='client.48814 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWrite_smithi016-74264-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:20.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:20 smithi183 ceph-mon[33745]: from='client.48817 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/0_smithi016-74514-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:20.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:20 smithi183 ceph-mon[33745]: osdmap e155: 8 total, 8 up, 8 in 2024-03-26T04:21:20.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:20 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3873220630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21"}]: dispatch 2024-03-26T04:21:20.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:20 smithi183 ceph-mon[33745]: from='client.48826 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21"}]: dispatch 2024-03-26T04:21:20.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:20 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3873220630' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21"}]: dispatch 2024-03-26T04:21:20.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:20 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:20.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:20 smithi183 ceph-mon[33745]: from='client.48826 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21"}]: dispatch 2024-03-26T04:21:20.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:20 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3873220630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:20.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:20 smithi183 ceph-mon[33745]: from='client.48826 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:21.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:21 smithi016 ceph-mon[29101]: from='client.48826 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:21:21.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:21 smithi016 ceph-mon[29101]: osdmap e156: 8 total, 8 up, 8 in 2024-03-26T04:21:21.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:21 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3873220630' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21"}]: dispatch 2024-03-26T04:21:21.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:21 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3910490755' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi016-74285-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:21.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:21 smithi016 ceph-mon[29101]: from='client.48826 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21"}]: dispatch 2024-03-26T04:21:21.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:21 smithi016 ceph-mon[29101]: from='client.48829 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi016-74285-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:21.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:21 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:21.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:21 smithi016 ceph-mon[37215]: from='client.48826 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:21:21.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:21 smithi016 ceph-mon[37215]: osdmap e156: 8 total, 8 up, 8 in 2024-03-26T04:21:21.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:21 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3873220630' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21"}]: dispatch 2024-03-26T04:21:21.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:21 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3910490755' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi016-74285-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:21.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:21 smithi016 ceph-mon[37215]: from='client.48826 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21"}]: dispatch 2024-03-26T04:21:21.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:21 smithi016 ceph-mon[37215]: from='client.48829 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi016-74285-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:21.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:21 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:21.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:21 smithi183 ceph-mon[33745]: from='client.48826 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:21:21.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:21 smithi183 ceph-mon[33745]: osdmap e156: 8 total, 8 up, 8 in 2024-03-26T04:21:21.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:21 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3873220630' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21"}]: dispatch 2024-03-26T04:21:21.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:21 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3910490755' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi016-74285-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:21.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:21 smithi183 ceph-mon[33745]: from='client.48826 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21"}]: dispatch 2024-03-26T04:21:21.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:21 smithi183 ceph-mon[33745]: from='client.48829 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi016-74285-22","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:21.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:21 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:22.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:22 smithi016 ceph-mon[29101]: pgmap v194: 388 pgs: 32 unknown, 356 active+clean; 592 KiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:21:22.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:22 smithi016 ceph-mon[29101]: from='client.48829 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi016-74285-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:22.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:22 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:22.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:22 smithi016 ceph-mon[29101]: osdmap e157: 8 total, 8 up, 8 in 2024-03-26T04:21:22.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:22 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1140963757' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi016-74264-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:22.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:22 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2310478084' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi016-74514-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:22.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:22 smithi016 ceph-mon[29101]: from='client.48838 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi016-74264-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:22.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:22 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:21:22.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:22 smithi016 ceph-mon[37215]: pgmap v194: 388 pgs: 32 unknown, 356 active+clean; 592 KiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:21:22.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:22 smithi016 ceph-mon[37215]: from='client.48829 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi016-74285-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:22.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:22 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:22.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:22 smithi016 ceph-mon[37215]: osdmap e157: 8 total, 8 up, 8 in 2024-03-26T04:21:22.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:22 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1140963757' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi016-74264-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:22.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:22 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2310478084' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi016-74514-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:22.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:22 smithi016 ceph-mon[37215]: from='client.48838 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi016-74264-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:22.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:22 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:21:22.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:21:22 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:21:22.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:22 smithi183 ceph-mon[33745]: pgmap v194: 388 pgs: 32 unknown, 356 active+clean; 592 KiB data, 596 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:21:22.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:22 smithi183 ceph-mon[33745]: from='client.48829 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime_smithi016-74285-22","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:22.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:22 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:22.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:22 smithi183 ceph-mon[33745]: osdmap e157: 8 total, 8 up, 8 in 2024-03-26T04:21:22.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:22 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1140963757' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi016-74264-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:22.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:22 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2310478084' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi016-74514-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:22.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:22 smithi183 ceph-mon[33745]: from='client.48838 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi016-74264-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:22.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:22 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:21:23.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:23 smithi016 ceph-mon[37215]: from='client.48826 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21"}]': finished 2024-03-26T04:21:23.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:23 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2310478084' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi016-74514-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:23.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:23 smithi016 ceph-mon[37215]: from='client.48838 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi016-74264-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:23.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:23 smithi016 ceph-mon[37215]: osdmap e158: 8 total, 8 up, 8 in 2024-03-26T04:21:23.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:23 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:21:23.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:23 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:23.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:23 smithi016 ceph-mon[29101]: from='client.48826 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21"}]': finished 2024-03-26T04:21:23.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:23 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2310478084' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi016-74514-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:23.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:23 smithi016 ceph-mon[29101]: from='client.48838 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi016-74264-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:23.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:23 smithi016 ceph-mon[29101]: osdmap e158: 8 total, 8 up, 8 in 2024-03-26T04:21:23.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:23 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:21:23.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:23 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:23.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:23 smithi183 ceph-mon[33745]: from='client.48826 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21"}]': finished 2024-03-26T04:21:23.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:23 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2310478084' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/1_smithi016-74514-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:23.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:23 smithi183 ceph-mon[33745]: from='client.48838 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlock_smithi016-74264-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:23.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:23 smithi183 ceph-mon[33745]: osdmap e158: 8 total, 8 up, 8 in 2024-03-26T04:21:23.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:23 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:21:23.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:23 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:24.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:24 smithi016 ceph-mon[37215]: pgmap v197: 428 pgs: 46 creating+peering, 4 unknown, 378 active+clean; 596 KiB data, 584 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-26T04:21:24.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:24 smithi016 ceph-mon[37215]: osdmap e159: 8 total, 8 up, 8 in 2024-03-26T04:21:24.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:24 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2109866103' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi016-74285-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:24.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:24 smithi016 ceph-mon[37215]: from='client.56318 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi016-74285-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:24.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:24 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1178551664' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi016-74264-27"}]: dispatch 2024-03-26T04:21:24.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:24 smithi016 ceph-mon[37215]: from='client.56327 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi016-74264-27"}]: dispatch 2024-03-26T04:21:24.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:24 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1178551664' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi016-74264-27"}]: dispatch 2024-03-26T04:21:24.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:24 smithi016 ceph-mon[37215]: from='client.56327 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi016-74264-27"}]: dispatch 2024-03-26T04:21:24.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:24 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1178551664' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi016-74264-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:24.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:24 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:24.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:24 smithi016 ceph-mon[37215]: from='client.56327 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi016-74264-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:24.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:24 smithi016 ceph-mon[29101]: pgmap v197: 428 pgs: 46 creating+peering, 4 unknown, 378 active+clean; 596 KiB data, 584 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-26T04:21:24.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:24 smithi016 ceph-mon[29101]: osdmap e159: 8 total, 8 up, 8 in 2024-03-26T04:21:24.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:24 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2109866103' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi016-74285-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:24.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:24 smithi016 ceph-mon[29101]: from='client.56318 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi016-74285-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:24.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:24 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1178551664' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi016-74264-27"}]: dispatch 2024-03-26T04:21:24.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:24 smithi016 ceph-mon[29101]: from='client.56327 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi016-74264-27"}]: dispatch 2024-03-26T04:21:24.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:24 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1178551664' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi016-74264-27"}]: dispatch 2024-03-26T04:21:24.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:24 smithi016 ceph-mon[29101]: from='client.56327 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi016-74264-27"}]: dispatch 2024-03-26T04:21:24.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:24 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1178551664' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi016-74264-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:24.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:24 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:24.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:24 smithi016 ceph-mon[29101]: from='client.56327 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi016-74264-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:24.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:24 smithi183 ceph-mon[33745]: pgmap v197: 428 pgs: 46 creating+peering, 4 unknown, 378 active+clean; 596 KiB data, 584 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-26T04:21:24.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:24 smithi183 ceph-mon[33745]: osdmap e159: 8 total, 8 up, 8 in 2024-03-26T04:21:24.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:24 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2109866103' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi016-74285-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:24.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:24 smithi183 ceph-mon[33745]: from='client.56318 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi016-74285-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:24.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:24 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1178551664' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi016-74264-27"}]: dispatch 2024-03-26T04:21:24.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:24 smithi183 ceph-mon[33745]: from='client.56327 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi016-74264-27"}]: dispatch 2024-03-26T04:21:24.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:24 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1178551664' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi016-74264-27"}]: dispatch 2024-03-26T04:21:24.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:24 smithi183 ceph-mon[33745]: from='client.56327 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi016-74264-27"}]: dispatch 2024-03-26T04:21:24.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:24 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1178551664' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi016-74264-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:24.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:24 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:24.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:24 smithi183 ceph-mon[33745]: from='client.56327 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi016-74264-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:25.377 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [ OK ] LibRadosSnaps snapshots: Running main() from gmock_main.cc 2024-03-26T04:21:25.378 INFO:tasks.workunit.client.0.smithi016.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2024-03-26T04:21:25.378 INFO:tasks.workunit.client.0.smithi016.stdout: snapshots: [----------] Global test environment set-up. 2024-03-26T04:21:25.378 INFO:tasks.workunit.client.0.smithi016.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2024-03-26T04:21:25.378 INFO:tasks.workunit.client.0.smithi016.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2024-03-26T04:21:25.378 INFO:tasks.workunit.client.0.smithi016.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (17988 ms) 2024-03-26T04:21:25.378 INFO:tasks.workunit.client.0.smithi016.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2024-03-26T04:21:25.378 INFO:tasks.workunit.client.0.smithi016.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (7374 ms) 2024-03-26T04:21:25.378 INFO:tasks.workunit.client.0.smithi016.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2024-03-26T04:21:25.378 INFO:tasks.workunit.client.0.smithi016.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (4292 ms) 2024-03-26T04:21:25.378 INFO:tasks.workunit.client.0.smithi016.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2024-03-26T04:21:25.378 INFO:tasks.workunit.client.0.smithi016.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (8215 ms) 2024-03-26T04:21:25.378 INFO:tasks.workunit.client.0.smithi016.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2024-03-26T04:21:25.378 INFO:tasks.workunit.client.0.smithi016.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (8281 ms) 2024-03-26T04:21:25.379 INFO:tasks.workunit.client.0.smithi016.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (46150 ms total) 2024-03-26T04:21:25.379 INFO:tasks.workunit.client.0.smithi016.stdout: snapshots: 2024-03-26T04:21:25.379 INFO:tasks.workunit.client.0.smithi016.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2024-03-26T04:21:25.379 INFO:tasks.workunit.client.0.smithi016.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2024-03-26T04:21:25.379 INFO:tasks.workunit.client.0.smithi016.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (5926 ms) 2024-03-26T04:21:25.379 INFO:tasks.workunit.client.0.smithi016.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2024-03-26T04:21:25.379 INFO:tasks.workunit.client.0.smithi016.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (6010 ms) 2024-03-26T04:21:25.379 INFO:tasks.workunit.client.0.smithi016.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2024-03-26T04:21:25.379 INFO:tasks.workunit.client.0.smithi016.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (8005 ms) 2024-03-26T04:21:25.379 INFO:tasks.workunit.client.0.smithi016.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2024-03-26T04:21:25.379 INFO:tasks.workunit.client.0.smithi016.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (6003 ms) 2024-03-26T04:21:25.379 INFO:tasks.workunit.client.0.smithi016.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2024-03-26T04:21:25.379 INFO:tasks.workunit.client.0.smithi016.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (4017 ms) 2024-03-26T04:21:25.379 INFO:tasks.workunit.client.0.smithi016.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2024-03-26T04:21:25.379 INFO:tasks.workunit.client.0.smithi016.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi016-75668-11. 2024-03-26T04:21:25.380 INFO:tasks.workunit.client.0.smithi016.stdout: snapshots: Waiting for snaps to purge. 2024-03-26T04:21:25.380 INFO:tasks.workunit.client.0.smithi016.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (20279 ms) 2024-03-26T04:21:25.380 INFO:tasks.workunit.client.0.smithi016.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (50240 ms total) 2024-03-26T04:21:25.380 INFO:tasks.workunit.client.0.smithi016.stdout: snapshots: 2024-03-26T04:21:25.380 INFO:tasks.workunit.client.0.smithi016.stdout: snapshots: [----------] Global test environment tear-down 2024-03-26T04:21:25.380 INFO:tasks.workunit.client.0.smithi016.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (96390 ms total) 2024-03-26T04:21:25.380 INFO:tasks.workunit.client.0.smithi016.stdout: snapshots: [ PASSED ] 11 tests. 2024-03-26T04:21:25.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:25 smithi016 ceph-mon[29101]: from='client.56318 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi016-74285-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:25.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:25 smithi016 ceph-mon[29101]: from='client.56327 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi016-74264-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:21:25.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:25 smithi016 ceph-mon[29101]: osdmap e160: 8 total, 8 up, 8 in 2024-03-26T04:21:25.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:25 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1178551664' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi016-74264-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi016-74264-27"}]: dispatch 2024-03-26T04:21:25.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:25 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/4224930591' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi016-74514-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:25.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:25 smithi016 ceph-mon[29101]: from='client.56327 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi016-74264-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi016-74264-27"}]: dispatch 2024-03-26T04:21:25.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:25 smithi016 ceph-mon[29101]: from='client.48856 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi016-74514-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:25.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:25 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:25.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:25 smithi016 ceph-mon[29101]: from='client.48856 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi016-74514-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:25.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:25 smithi016 ceph-mon[37215]: from='client.56318 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi016-74285-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:25.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:25 smithi016 ceph-mon[37215]: from='client.56327 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi016-74264-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:21:25.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:25 smithi016 ceph-mon[37215]: osdmap e160: 8 total, 8 up, 8 in 2024-03-26T04:21:25.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:25 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1178551664' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi016-74264-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi016-74264-27"}]: dispatch 2024-03-26T04:21:25.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:25 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/4224930591' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi016-74514-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:25.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:25 smithi016 ceph-mon[37215]: from='client.56327 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi016-74264-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi016-74264-27"}]: dispatch 2024-03-26T04:21:25.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:25 smithi016 ceph-mon[37215]: from='client.48856 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi016-74514-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:25.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:25 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:25.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:25 smithi016 ceph-mon[37215]: from='client.48856 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi016-74514-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:25.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:25 smithi183 ceph-mon[33745]: from='client.56318 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OperateMtime2_smithi016-74285-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:25.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:25 smithi183 ceph-mon[33745]: from='client.56327 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWrite_smithi016-74264-27", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:21:25.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:25 smithi183 ceph-mon[33745]: osdmap e160: 8 total, 8 up, 8 in 2024-03-26T04:21:25.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:25 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1178551664' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi016-74264-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi016-74264-27"}]: dispatch 2024-03-26T04:21:25.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:25 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/4224930591' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi016-74514-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:25.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:25 smithi183 ceph-mon[33745]: from='client.56327 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi016-74264-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi016-74264-27"}]: dispatch 2024-03-26T04:21:25.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:25 smithi183 ceph-mon[33745]: from='client.48856 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi016-74514-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:25.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:25 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:25.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:25 smithi183 ceph-mon[33745]: from='client.48856 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosChecksum/2_smithi016-74514-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:26.383 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:26 smithi183 ceph-mon[33745]: pgmap v200: 428 pgs: 4 creating+peering, 68 unknown, 356 active+clean; 592 KiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:21:26.383 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:26 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:26.383 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:26 smithi183 ceph-mon[33745]: osdmap e161: 8 total, 8 up, 8 in 2024-03-26T04:21:26.383 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:26 smithi183 ceph-mon[33745]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:21:26.383 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:26 smithi183 ceph-mon[33745]: from='client.56327 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi016-74264-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi016-74264-27"}]': finished 2024-03-26T04:21:26.383 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:26 smithi183 ceph-mon[33745]: osdmap e162: 8 total, 8 up, 8 in 2024-03-26T04:21:26.383 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:26 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/4058123619' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi016-74285-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:26.383 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:26 smithi183 ceph-mon[33745]: from='client.56333 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi016-74285-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:26.384 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:26 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3719561063' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi016-74514-36"}]: dispatch 2024-03-26T04:21:26.384 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:26 smithi183 ceph-mon[33745]: from='client.56339 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi016-74514-36"}]: dispatch 2024-03-26T04:21:26.384 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:26 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3719561063' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi016-74514-36"}]: dispatch 2024-03-26T04:21:26.384 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:26 smithi183 ceph-mon[33745]: from='client.56339 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi016-74514-36"}]: dispatch 2024-03-26T04:21:26.384 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:26 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3719561063' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi016-74514-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:26.384 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:26 smithi183 ceph-mon[33745]: from='client.56339 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi016-74514-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:26.384 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:26 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:26.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:26 smithi016 ceph-mon[37215]: pgmap v200: 428 pgs: 4 creating+peering, 68 unknown, 356 active+clean; 592 KiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:21:26.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:26 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:26.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:26 smithi016 ceph-mon[37215]: osdmap e161: 8 total, 8 up, 8 in 2024-03-26T04:21:26.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:26 smithi016 ceph-mon[37215]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:21:26.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:26 smithi016 ceph-mon[37215]: from='client.56327 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi016-74264-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi016-74264-27"}]': finished 2024-03-26T04:21:26.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:26 smithi016 ceph-mon[37215]: osdmap e162: 8 total, 8 up, 8 in 2024-03-26T04:21:26.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:26 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/4058123619' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi016-74285-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:26.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:26 smithi016 ceph-mon[37215]: from='client.56333 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi016-74285-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:26.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:26 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3719561063' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi016-74514-36"}]: dispatch 2024-03-26T04:21:26.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:26 smithi016 ceph-mon[37215]: from='client.56339 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi016-74514-36"}]: dispatch 2024-03-26T04:21:26.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:26 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3719561063' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi016-74514-36"}]: dispatch 2024-03-26T04:21:26.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:26 smithi016 ceph-mon[37215]: from='client.56339 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi016-74514-36"}]: dispatch 2024-03-26T04:21:26.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:26 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3719561063' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi016-74514-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:26.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:26 smithi016 ceph-mon[37215]: from='client.56339 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi016-74514-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:26.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:26 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:26.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:26 smithi016 ceph-mon[29101]: pgmap v200: 428 pgs: 4 creating+peering, 68 unknown, 356 active+clean; 592 KiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:21:26.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:26 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:26.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:26 smithi016 ceph-mon[29101]: osdmap e161: 8 total, 8 up, 8 in 2024-03-26T04:21:26.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:26 smithi016 ceph-mon[29101]: Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:21:26.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:26 smithi016 ceph-mon[29101]: from='client.56327 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWrite_smithi016-74264-27", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWrite_smithi016-74264-27"}]': finished 2024-03-26T04:21:26.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:26 smithi016 ceph-mon[29101]: osdmap e162: 8 total, 8 up, 8 in 2024-03-26T04:21:26.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:26 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/4058123619' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi016-74285-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:26.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:26 smithi016 ceph-mon[29101]: from='client.56333 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi016-74285-24","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:26.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:26 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3719561063' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi016-74514-36"}]: dispatch 2024-03-26T04:21:26.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:26 smithi016 ceph-mon[29101]: from='client.56339 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi016-74514-36"}]: dispatch 2024-03-26T04:21:26.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:26 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3719561063' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi016-74514-36"}]: dispatch 2024-03-26T04:21:26.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:26 smithi016 ceph-mon[29101]: from='client.56339 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi016-74514-36"}]: dispatch 2024-03-26T04:21:26.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:26 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3719561063' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi016-74514-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:26.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:26 smithi016 ceph-mon[29101]: from='client.56339 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi016-74514-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:26.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:26 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:28.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:28 smithi016 ceph-mon[29101]: pgmap v203: 372 pgs: 10 creating+peering, 38 unknown, 324 active+clean; 592 KiB data, 584 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:21:28.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:28 smithi016 ceph-mon[29101]: from='client.56333 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi016-74285-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:28.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:28 smithi016 ceph-mon[29101]: from='client.56339 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi016-74514-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:21:28.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:28 smithi016 ceph-mon[29101]: osdmap e163: 8 total, 8 up, 8 in 2024-03-26T04:21:28.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:28 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3719561063' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi016-74514-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi016-74514-36"}]: dispatch 2024-03-26T04:21:28.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:28 smithi016 ceph-mon[29101]: from='client.56339 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi016-74514-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi016-74514-36"}]: dispatch 2024-03-26T04:21:28.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:28 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:28.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:28 smithi016 ceph-mon[37215]: pgmap v203: 372 pgs: 10 creating+peering, 38 unknown, 324 active+clean; 592 KiB data, 584 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:21:28.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:28 smithi016 ceph-mon[37215]: from='client.56333 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi016-74285-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:28.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:28 smithi016 ceph-mon[37215]: from='client.56339 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi016-74514-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:21:28.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:28 smithi016 ceph-mon[37215]: osdmap e163: 8 total, 8 up, 8 in 2024-03-26T04:21:28.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:28 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3719561063' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi016-74514-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi016-74514-36"}]: dispatch 2024-03-26T04:21:28.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:28 smithi016 ceph-mon[37215]: from='client.56339 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi016-74514-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi016-74514-36"}]: dispatch 2024-03-26T04:21:28.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:28 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:28.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:28 smithi183 ceph-mon[33745]: pgmap v203: 372 pgs: 10 creating+peering, 38 unknown, 324 active+clean; 592 KiB data, 584 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:21:28.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:28 smithi183 ceph-mon[33745]: from='client.56333 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "StatRemovePP_smithi016-74285-24","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:28.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:28 smithi183 ceph-mon[33745]: from='client.56339 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosMiscECPP_smithi016-74514-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:21:28.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:28 smithi183 ceph-mon[33745]: osdmap e163: 8 total, 8 up, 8 in 2024-03-26T04:21:28.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:28 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3719561063' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi016-74514-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi016-74514-36"}]: dispatch 2024-03-26T04:21:28.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:28 smithi183 ceph-mon[33745]: from='client.56339 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi016-74514-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi016-74514-36"}]: dispatch 2024-03-26T04:21:28.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:28 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:29.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:21:28 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:21:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:21:29.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:29 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:29.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:29 smithi016 ceph-mon[29101]: osdmap e164: 8 total, 8 up, 8 in 2024-03-26T04:21:29.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:29 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1178551664' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi016-74264-27"}]: dispatch 2024-03-26T04:21:29.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:29 smithi016 ceph-mon[29101]: from='client.56327 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi016-74264-27"}]: dispatch 2024-03-26T04:21:29.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:29 smithi016 ceph-mon[29101]: from='client.56339 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi016-74514-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi016-74514-36"}]': finished 2024-03-26T04:21:29.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:29 smithi016 ceph-mon[29101]: from='client.56327 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi016-74264-27"}]': finished 2024-03-26T04:21:29.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:29 smithi016 ceph-mon[29101]: osdmap e165: 8 total, 8 up, 8 in 2024-03-26T04:21:29.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:29 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1178551664' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi016-74264-27"}]: dispatch 2024-03-26T04:21:29.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:29 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2100599286' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi016-74285-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:29.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:29 smithi016 ceph-mon[29101]: from='client.56327 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi016-74264-27"}]: dispatch 2024-03-26T04:21:29.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:29 smithi016 ceph-mon[29101]: from='client.56345 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi016-74285-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:29.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:29 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:29.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:29 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:29.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:29 smithi016 ceph-mon[37215]: osdmap e164: 8 total, 8 up, 8 in 2024-03-26T04:21:29.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:29 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1178551664' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi016-74264-27"}]: dispatch 2024-03-26T04:21:29.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:29 smithi016 ceph-mon[37215]: from='client.56327 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi016-74264-27"}]: dispatch 2024-03-26T04:21:29.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:29 smithi016 ceph-mon[37215]: from='client.56339 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi016-74514-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi016-74514-36"}]': finished 2024-03-26T04:21:29.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:29 smithi016 ceph-mon[37215]: from='client.56327 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi016-74264-27"}]': finished 2024-03-26T04:21:29.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:29 smithi016 ceph-mon[37215]: osdmap e165: 8 total, 8 up, 8 in 2024-03-26T04:21:29.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:29 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1178551664' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi016-74264-27"}]: dispatch 2024-03-26T04:21:29.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:29 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2100599286' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi016-74285-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:29.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:29 smithi016 ceph-mon[37215]: from='client.56327 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi016-74264-27"}]: dispatch 2024-03-26T04:21:29.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:29 smithi016 ceph-mon[37215]: from='client.56345 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi016-74285-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:29.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:29 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:29.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:29 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:29.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:29 smithi183 ceph-mon[33745]: osdmap e164: 8 total, 8 up, 8 in 2024-03-26T04:21:29.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:29 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1178551664' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi016-74264-27"}]: dispatch 2024-03-26T04:21:29.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:29 smithi183 ceph-mon[33745]: from='client.56327 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi016-74264-27"}]: dispatch 2024-03-26T04:21:29.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:29 smithi183 ceph-mon[33745]: from='client.56339 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosMiscECPP_smithi016-74514-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosMiscECPP_smithi016-74514-36"}]': finished 2024-03-26T04:21:29.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:29 smithi183 ceph-mon[33745]: from='client.56327 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWrite_smithi016-74264-27"}]': finished 2024-03-26T04:21:29.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:29 smithi183 ceph-mon[33745]: osdmap e165: 8 total, 8 up, 8 in 2024-03-26T04:21:29.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:29 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1178551664' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi016-74264-27"}]: dispatch 2024-03-26T04:21:29.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:29 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2100599286' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi016-74285-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:29.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:29 smithi183 ceph-mon[33745]: from='client.56327 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi016-74264-27"}]: dispatch 2024-03-26T04:21:29.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:29 smithi183 ceph-mon[33745]: from='client.56345 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi016-74285-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:29.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:29 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:30.120 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: Running main() from gmock_main.cc 2024-03-26T04:21:30.120 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-03-26T04:21:30.120 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [----------] Global test environment set-up. 2024-03-26T04:21:30.120 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-03-26T04:21:30.120 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-03-26T04:21:30.120 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ OK ] LibRadosAio.TooBig (10358 ms) 2024-03-26T04:21:30.120 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-03-26T04:21:30.120 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (3914 ms) 2024-03-26T04:21:30.120 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-03-26T04:21:30.120 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (8071 ms) 2024-03-26T04:21:30.120 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-03-26T04:21:30.120 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (4521 ms) 2024-03-26T04:21:30.121 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-03-26T04:21:30.121 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (3257 ms) 2024-03-26T04:21:30.121 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-03-26T04:21:30.121 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (3349 ms) 2024-03-26T04:21:30.121 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-03-26T04:21:30.121 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (3452 ms) 2024-03-26T04:21:30.121 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-03-26T04:21:30.121 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (3632 ms) 2024-03-26T04:21:30.121 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-03-26T04:21:30.121 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (2806 ms) 2024-03-26T04:21:30.121 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-03-26T04:21:30.121 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (3214 ms) 2024-03-26T04:21:30.121 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-03-26T04:21:30.121 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (2963 ms) 2024-03-26T04:21:30.121 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-03-26T04:21:30.122 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (2996 ms) 2024-03-26T04:21:30.122 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-03-26T04:21:30.122 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (3011 ms) 2024-03-26T04:21:30.122 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-03-26T04:21:30.122 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (2995 ms) 2024-03-26T04:21:30.122 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-03-26T04:21:30.122 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ OK ] LibRadosAio.Flush (3005 ms) 2024-03-26T04:21:30.122 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-03-26T04:21:30.122 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (2999 ms) 2024-03-26T04:21:30.122 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-03-26T04:21:30.122 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (2999 ms) 2024-03-26T04:21:30.122 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-03-26T04:21:30.122 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (3007 ms) 2024-03-26T04:21:30.123 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-03-26T04:21:30.123 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (3008 ms) 2024-03-26T04:21:30.123 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-03-26T04:21:30.123 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (3011 ms) 2024-03-26T04:21:30.123 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-03-26T04:21:30.123 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (3002 ms) 2024-03-26T04:21:30.123 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-03-26T04:21:30.123 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (3013 ms) 2024-03-26T04:21:30.123 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-03-26T04:21:30.123 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (3036 ms) 2024-03-26T04:21:30.123 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-03-26T04:21:30.123 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (2929 ms) 2024-03-26T04:21:30.123 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-03-26T04:21:30.123 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (3010 ms) 2024-03-26T04:21:30.124 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-03-26T04:21:30.124 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3023 ms) 2024-03-26T04:21:30.124 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [----------] 26 tests from LibRadosAio (94581 ms total) 2024-03-26T04:21:30.124 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: 2024-03-26T04:21:30.124 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-03-26T04:21:30.124 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWrite 2024-03-26T04:21:30.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:30 smithi016 ceph-mon[37215]: pgmap v206: 332 pgs: 332 active+clean; 616 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 9.7 KiB/s wr, 2 op/s 2024-03-26T04:21:30.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:30 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:21:30.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:30 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:21:30.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:30 smithi016 ceph-mon[37215]: from='client.56327 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi016-74264-27"}]': finished 2024-03-26T04:21:30.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:30 smithi016 ceph-mon[37215]: from='client.56345 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi016-74285-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:30.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:30 smithi016 ceph-mon[37215]: osdmap e166: 8 total, 8 up, 8 in 2024-03-26T04:21:30.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:30 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:21:30.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:30 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:21:30.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:30 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2588716529' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi016-74264-28"}]: dispatch 2024-03-26T04:21:30.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:30 smithi016 ceph-mon[37215]: from='client.48877 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi016-74264-28"}]: dispatch 2024-03-26T04:21:30.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:30 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2588716529' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi016-74264-28"}]: dispatch 2024-03-26T04:21:30.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:30 smithi016 ceph-mon[37215]: from='client.48877 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi016-74264-28"}]: dispatch 2024-03-26T04:21:30.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:30 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2588716529' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi016-74264-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:30.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:30 smithi016 ceph-mon[37215]: from='client.48877 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi016-74264-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:30.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:30 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:30.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:30 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi016-74374-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T04:21:30.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:30 smithi016 ceph-mon[37215]: from='client.24709 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi016-74374-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T04:21:30.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:30 smithi016 ceph-mon[29101]: pgmap v206: 332 pgs: 332 active+clean; 616 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 9.7 KiB/s wr, 2 op/s 2024-03-26T04:21:30.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:30 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:21:30.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:30 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:21:30.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:30 smithi016 ceph-mon[29101]: from='client.56327 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi016-74264-27"}]': finished 2024-03-26T04:21:30.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:30 smithi016 ceph-mon[29101]: from='client.56345 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi016-74285-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:30.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:30 smithi016 ceph-mon[29101]: osdmap e166: 8 total, 8 up, 8 in 2024-03-26T04:21:30.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:30 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:21:30.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:30 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:21:30.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:30 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2588716529' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi016-74264-28"}]: dispatch 2024-03-26T04:21:30.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:30 smithi016 ceph-mon[29101]: from='client.48877 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi016-74264-28"}]: dispatch 2024-03-26T04:21:30.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:30 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2588716529' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi016-74264-28"}]: dispatch 2024-03-26T04:21:30.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:30 smithi016 ceph-mon[29101]: from='client.48877 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi016-74264-28"}]: dispatch 2024-03-26T04:21:30.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:30 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2588716529' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi016-74264-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:30.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:30 smithi016 ceph-mon[29101]: from='client.48877 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi016-74264-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:30.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:30 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:30.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:30 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi016-74374-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T04:21:30.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:30 smithi016 ceph-mon[29101]: from='client.24709 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi016-74374-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T04:21:30.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:30 smithi183 ceph-mon[33745]: pgmap v206: 332 pgs: 332 active+clean; 616 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 9.7 KiB/s wr, 2 op/s 2024-03-26T04:21:30.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:30 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:21:30.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:30 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:21:30.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:30 smithi183 ceph-mon[33745]: from='client.56327 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWrite_smithi016-74264-27"}]': finished 2024-03-26T04:21:30.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:30 smithi183 ceph-mon[33745]: from='client.56345 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ExecuteClassPP_smithi016-74285-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:30.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:30 smithi183 ceph-mon[33745]: osdmap e166: 8 total, 8 up, 8 in 2024-03-26T04:21:30.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:30 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:21:30.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:30 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:21:30.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:30 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2588716529' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi016-74264-28"}]: dispatch 2024-03-26T04:21:30.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:30 smithi183 ceph-mon[33745]: from='client.48877 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi016-74264-28"}]: dispatch 2024-03-26T04:21:30.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:30 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2588716529' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi016-74264-28"}]: dispatch 2024-03-26T04:21:30.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:30 smithi183 ceph-mon[33745]: from='client.48877 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi016-74264-28"}]: dispatch 2024-03-26T04:21:30.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:30 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2588716529' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi016-74264-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:30.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:30 smithi183 ceph-mon[33745]: from='client.48877 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi016-74264-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:30.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:30 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:30.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:30 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi016-74374-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T04:21:30.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:30 smithi183 ceph-mon[33745]: from='client.24709 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi016-74374-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T04:21:31.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:31 smithi016 ceph-mon[29101]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:21:31.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:31 smithi016 ceph-mon[29101]: from='client.48877 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi016-74264-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:21:31.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:31 smithi016 ceph-mon[29101]: from='client.24709 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi016-74374-1","var":"pg_num","val":"11"}]': finished 2024-03-26T04:21:31.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:31 smithi016 ceph-mon[29101]: osdmap e167: 8 total, 8 up, 8 in 2024-03-26T04:21:31.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:31 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2588716529' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi016-74264-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi016-74264-28"}]: dispatch 2024-03-26T04:21:31.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:31 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:31.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:31 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3719561063' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi016-74514-36"}]: dispatch 2024-03-26T04:21:31.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:31 smithi016 ceph-mon[29101]: from='client.48877 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi016-74264-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi016-74264-28"}]: dispatch 2024-03-26T04:21:31.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:31 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:31.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:31 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi016-74374-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T04:21:31.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:31 smithi016 ceph-mon[29101]: from='client.56339 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi016-74514-36"}]: dispatch 2024-03-26T04:21:31.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:31 smithi016 ceph-mon[29101]: from='client.24709 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi016-74374-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T04:21:31.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:31 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:21:31.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:31 smithi016 ceph-mon[37215]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:21:31.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:31 smithi016 ceph-mon[37215]: from='client.48877 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi016-74264-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:21:31.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:31 smithi016 ceph-mon[37215]: from='client.24709 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi016-74374-1","var":"pg_num","val":"11"}]': finished 2024-03-26T04:21:31.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:31 smithi016 ceph-mon[37215]: osdmap e167: 8 total, 8 up, 8 in 2024-03-26T04:21:31.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:31 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2588716529' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi016-74264-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi016-74264-28"}]: dispatch 2024-03-26T04:21:31.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:31 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:31.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:31 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3719561063' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi016-74514-36"}]: dispatch 2024-03-26T04:21:31.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:31 smithi016 ceph-mon[37215]: from='client.48877 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi016-74264-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi016-74264-28"}]: dispatch 2024-03-26T04:21:31.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:31 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:31.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:31 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi016-74374-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T04:21:31.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:31 smithi016 ceph-mon[37215]: from='client.56339 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi016-74514-36"}]: dispatch 2024-03-26T04:21:31.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:31 smithi016 ceph-mon[37215]: from='client.24709 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi016-74374-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T04:21:31.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:31 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:21:31.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:31 smithi183 ceph-mon[33745]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:21:31.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:31 smithi183 ceph-mon[33745]: from='client.48877 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForComplete_smithi016-74264-28", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:21:31.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:31 smithi183 ceph-mon[33745]: from='client.24709 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi016-74374-1","var":"pg_num","val":"11"}]': finished 2024-03-26T04:21:31.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:31 smithi183 ceph-mon[33745]: osdmap e167: 8 total, 8 up, 8 in 2024-03-26T04:21:31.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:31 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2588716529' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi016-74264-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi016-74264-28"}]: dispatch 2024-03-26T04:21:31.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:31 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:31.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:31 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3719561063' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi016-74514-36"}]: dispatch 2024-03-26T04:21:31.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:31 smithi183 ceph-mon[33745]: from='client.48877 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi016-74264-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi016-74264-28"}]: dispatch 2024-03-26T04:21:31.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:31 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:31.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:31 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi016-74374-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T04:21:31.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:31 smithi183 ceph-mon[33745]: from='client.56339 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi016-74514-36"}]: dispatch 2024-03-26T04:21:31.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:31 smithi183 ceph-mon[33745]: from='client.24709 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi016-74374-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T04:21:31.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:31 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:21:32.396 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:21:32 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:21:32.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:32 smithi016 ceph-mon[37215]: pgmap v209: 372 pgs: 40 unknown, 332 active+clean; 616 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 10 KiB/s wr, 3 op/s 2024-03-26T04:21:32.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:32 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:21:32.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:32 smithi016 ceph-mon[37215]: from='client.56339 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi016-74514-36"}]': finished 2024-03-26T04:21:32.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:32 smithi016 ceph-mon[37215]: from='client.24709 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi016-74374-1","var":"pgp_num","val":"11"}]': finished 2024-03-26T04:21:32.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:32 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:21:32.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:32 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:32.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:32 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-13"}]: dispatch 2024-03-26T04:21:32.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:32 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3719561063' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi016-74514-36"}]: dispatch 2024-03-26T04:21:32.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:32 smithi016 ceph-mon[37215]: osdmap e168: 8 total, 8 up, 8 in 2024-03-26T04:21:32.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:32 smithi016 ceph-mon[37215]: from='client.56339 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi016-74514-36"}]: dispatch 2024-03-26T04:21:32.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:32 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-13"}]: dispatch 2024-03-26T04:21:32.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:32 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2547889520' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi016-74285-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:32.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:32 smithi016 ceph-mon[29101]: pgmap v209: 372 pgs: 40 unknown, 332 active+clean; 616 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 10 KiB/s wr, 3 op/s 2024-03-26T04:21:32.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:32 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:21:32.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:32 smithi016 ceph-mon[29101]: from='client.56339 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi016-74514-36"}]': finished 2024-03-26T04:21:32.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:32 smithi016 ceph-mon[29101]: from='client.24709 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi016-74374-1","var":"pgp_num","val":"11"}]': finished 2024-03-26T04:21:32.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:32 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:21:32.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:32 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:32.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:32 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-13"}]: dispatch 2024-03-26T04:21:32.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:32 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3719561063' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi016-74514-36"}]: dispatch 2024-03-26T04:21:32.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:32 smithi016 ceph-mon[29101]: osdmap e168: 8 total, 8 up, 8 in 2024-03-26T04:21:32.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:32 smithi016 ceph-mon[29101]: from='client.56339 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi016-74514-36"}]: dispatch 2024-03-26T04:21:32.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:32 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-13"}]: dispatch 2024-03-26T04:21:32.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:32 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2547889520' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi016-74285-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:32.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:32 smithi183 ceph-mon[33745]: pgmap v209: 372 pgs: 40 unknown, 332 active+clean; 616 KiB data, 577 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 10 KiB/s wr, 3 op/s 2024-03-26T04:21:32.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:32 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:21:32.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:32 smithi183 ceph-mon[33745]: from='client.56339 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosMiscECPP_smithi016-74514-36"}]': finished 2024-03-26T04:21:32.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:32 smithi183 ceph-mon[33745]: from='client.24709 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi016-74374-1","var":"pgp_num","val":"11"}]': finished 2024-03-26T04:21:32.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:32 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:21:32.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:32 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:32.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:32 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-13"}]: dispatch 2024-03-26T04:21:32.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:32 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3719561063' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi016-74514-36"}]: dispatch 2024-03-26T04:21:32.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:32 smithi183 ceph-mon[33745]: osdmap e168: 8 total, 8 up, 8 in 2024-03-26T04:21:32.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:32 smithi183 ceph-mon[33745]: from='client.56339 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi016-74514-36"}]: dispatch 2024-03-26T04:21:32.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:32 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-13"}]: dispatch 2024-03-26T04:21:32.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:32 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2547889520' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "OmapPP_smithi016-74285-26","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:33.088 INFO:tasks.workunit.client.0.smithi016.stdout:(67854 ms total) 2024-03-26T04:21:33.089 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: 2024-03-26T04:21:33.089 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-03-26T04:21:33.089 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-03-26T04:21:33.089 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (119 ms) 2024-03-26T04:21:33.089 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (119 ms total) 2024-03-26T04:21:33.089 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: 2024-03-26T04:21:33.089 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-03-26T04:21:33.089 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-03-26T04:21:33.089 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (55 ms) 2024-03-26T04:21:33.089 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-03-26T04:21:33.089 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (2 ms) 2024-03-26T04:21:33.090 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (57 ms total) 2024-03-26T04:21:33.090 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: 2024-03-26T04:21:33.090 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-03-26T04:21:33.090 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-03-26T04:21:33.090 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (55 ms) 2024-03-26T04:21:33.090 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-03-26T04:21:33.090 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (4 ms) 2024-03-26T04:21:33.090 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (59 ms total) 2024-03-26T04:21:33.090 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: 2024-03-26T04:21:33.090 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-03-26T04:21:33.090 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-03-26T04:21:33.090 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (34 ms) 2024-03-26T04:21:33.090 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-03-26T04:21:33.091 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (10 ms) 2024-03-26T04:21:33.091 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (44 ms total) 2024-03-26T04:21:33.091 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: 2024-03-26T04:21:33.091 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-03-26T04:21:33.091 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-03-26T04:21:33.091 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1054 ms) 2024-03-26T04:21:33.091 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1054 ms total) 2024-03-26T04:21:33.091 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: 2024-03-26T04:21:33.091 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-03-26T04:21:33.091 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (104445 ms total) 2024-03-26T04:21:33.091 INFO:tasks.workunit.client.0.smithi016.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-03-26T04:21:33.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:33 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:21:33.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:33 smithi016 ceph-mon[37215]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:21:33.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:33 smithi016 ceph-mon[37215]: from='client.48877 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi016-74264-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi016-74264-28"}]': finished 2024-03-26T04:21:33.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:33 smithi016 ceph-mon[37215]: from='client.56339 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi016-74514-36"}]': finished 2024-03-26T04:21:33.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:33 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-13"}]': finished 2024-03-26T04:21:33.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:33 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2547889520' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi016-74285-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:33.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:33 smithi016 ceph-mon[37215]: osdmap e169: 8 total, 8 up, 8 in 2024-03-26T04:21:33.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:33 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:33.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:33 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:21:33.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:33 smithi016 ceph-mon[29101]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:21:33.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:33 smithi016 ceph-mon[29101]: from='client.48877 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi016-74264-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi016-74264-28"}]': finished 2024-03-26T04:21:33.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:33 smithi016 ceph-mon[29101]: from='client.56339 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi016-74514-36"}]': finished 2024-03-26T04:21:33.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:33 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-13"}]': finished 2024-03-26T04:21:33.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:33 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2547889520' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi016-74285-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:33.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:33 smithi016 ceph-mon[29101]: osdmap e169: 8 total, 8 up, 8 in 2024-03-26T04:21:33.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:33 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:33.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:33 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:21:33.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:33 smithi183 ceph-mon[33745]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:21:33.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:33 smithi183 ceph-mon[33745]: from='client.48877 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForComplete_smithi016-74264-28", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForComplete_smithi016-74264-28"}]': finished 2024-03-26T04:21:33.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:33 smithi183 ceph-mon[33745]: from='client.56339 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosMiscECPP_smithi016-74514-36"}]': finished 2024-03-26T04:21:33.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:33 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-13"}]': finished 2024-03-26T04:21:33.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:33 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2547889520' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "OmapPP_smithi016-74285-26","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:33.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:33 smithi183 ceph-mon[33745]: osdmap e169: 8 total, 8 up, 8 in 2024-03-26T04:21:33.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:33 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:34.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:34 smithi016 ceph-mon[37215]: pgmap v212: 364 pgs: 32 unknown, 332 active+clean; 617 KiB data, 575 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.0 KiB/s wr, 22 op/s 2024-03-26T04:21:34.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:34 smithi016 ceph-mon[37215]: osdmap e170: 8 total, 8 up, 8 in 2024-03-26T04:21:34.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:34 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:34.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:34 smithi016 ceph-mon[29101]: pgmap v212: 364 pgs: 32 unknown, 332 active+clean; 617 KiB data, 575 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.0 KiB/s wr, 22 op/s 2024-03-26T04:21:34.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:34 smithi016 ceph-mon[29101]: osdmap e170: 8 total, 8 up, 8 in 2024-03-26T04:21:34.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:34 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:35.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:34 smithi183 ceph-mon[33745]: pgmap v212: 364 pgs: 32 unknown, 332 active+clean; 617 KiB data, 575 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.0 KiB/s wr, 22 op/s 2024-03-26T04:21:35.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:34 smithi183 ceph-mon[33745]: osdmap e170: 8 total, 8 up, 8 in 2024-03-26T04:21:35.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:34 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:36.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:36 smithi016 ceph-mon[29101]: pgmap v215: 308 pgs: 8 unknown, 300 active+clean; 617 KiB data, 575 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s wr, 0 op/s 2024-03-26T04:21:36.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:36 smithi016 ceph-mon[29101]: osdmap e171: 8 total, 8 up, 8 in 2024-03-26T04:21:36.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:36 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:36.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:36 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2588716529' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi016-74264-28"}]: dispatch 2024-03-26T04:21:36.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:36 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1332200885' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi016-74285-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:36.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:36 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:36.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:36 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3873220630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21"}]: dispatch 2024-03-26T04:21:36.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:36 smithi016 ceph-mon[29101]: from='client.48877 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi016-74264-28"}]: dispatch 2024-03-26T04:21:36.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:36 smithi016 ceph-mon[29101]: from='client.48889 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi016-74285-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:36.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:36 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:36.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:36 smithi016 ceph-mon[29101]: from='client.48826 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21"}]: dispatch 2024-03-26T04:21:36.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:36 smithi016 ceph-mon[29101]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:21:36.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:36 smithi016 ceph-mon[29101]: from='client.48877 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi016-74264-28"}]': finished 2024-03-26T04:21:36.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:36 smithi016 ceph-mon[29101]: from='client.48889 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi016-74285-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:36.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:36 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:36.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:36 smithi016 ceph-mon[29101]: from='client.48826 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21"}]': finished 2024-03-26T04:21:36.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:36 smithi016 ceph-mon[29101]: osdmap e172: 8 total, 8 up, 8 in 2024-03-26T04:21:36.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:36 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2588716529' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi016-74264-28"}]: dispatch 2024-03-26T04:21:36.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:36 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3873220630' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21"}]: dispatch 2024-03-26T04:21:36.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:36 smithi016 ceph-mon[29101]: from='client.48877 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi016-74264-28"}]: dispatch 2024-03-26T04:21:36.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:36 smithi016 ceph-mon[37215]: pgmap v215: 308 pgs: 8 unknown, 300 active+clean; 617 KiB data, 575 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s wr, 0 op/s 2024-03-26T04:21:36.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:36 smithi016 ceph-mon[37215]: osdmap e171: 8 total, 8 up, 8 in 2024-03-26T04:21:36.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:36 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:36.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:36 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2588716529' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi016-74264-28"}]: dispatch 2024-03-26T04:21:36.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:36 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1332200885' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi016-74285-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:36.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:36 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:36.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:36 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3873220630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21"}]: dispatch 2024-03-26T04:21:36.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:36 smithi016 ceph-mon[37215]: from='client.48877 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi016-74264-28"}]: dispatch 2024-03-26T04:21:36.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:36 smithi016 ceph-mon[37215]: from='client.48889 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi016-74285-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:36.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:36 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:36.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:36 smithi016 ceph-mon[37215]: from='client.48826 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21"}]: dispatch 2024-03-26T04:21:36.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:36 smithi016 ceph-mon[37215]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:21:36.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:36 smithi016 ceph-mon[37215]: from='client.48877 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi016-74264-28"}]': finished 2024-03-26T04:21:36.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:36 smithi016 ceph-mon[37215]: from='client.48889 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi016-74285-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:36.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:36 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:36.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:36 smithi016 ceph-mon[37215]: from='client.48826 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21"}]': finished 2024-03-26T04:21:36.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:36 smithi016 ceph-mon[37215]: osdmap e172: 8 total, 8 up, 8 in 2024-03-26T04:21:36.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:36 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2588716529' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi016-74264-28"}]: dispatch 2024-03-26T04:21:36.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:36 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3873220630' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21"}]: dispatch 2024-03-26T04:21:36.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:36 smithi016 ceph-mon[37215]: from='client.48877 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi016-74264-28"}]: dispatch 2024-03-26T04:21:36.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:36 smithi183 ceph-mon[33745]: pgmap v215: 308 pgs: 8 unknown, 300 active+clean; 617 KiB data, 575 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s wr, 0 op/s 2024-03-26T04:21:36.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:36 smithi183 ceph-mon[33745]: osdmap e171: 8 total, 8 up, 8 in 2024-03-26T04:21:36.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:36 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:36.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:36 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2588716529' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi016-74264-28"}]: dispatch 2024-03-26T04:21:36.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:36 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1332200885' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi016-74285-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:36.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:36 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:36.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:36 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3873220630' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21"}]: dispatch 2024-03-26T04:21:36.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:36 smithi183 ceph-mon[33745]: from='client.48877 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi016-74264-28"}]: dispatch 2024-03-26T04:21:36.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:36 smithi183 ceph-mon[33745]: from='client.48889 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi016-74285-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:36.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:36 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-15","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:36.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:36 smithi183 ceph-mon[33745]: from='client.48826 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21"}]: dispatch 2024-03-26T04:21:36.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:36 smithi183 ceph-mon[33745]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:21:36.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:36 smithi183 ceph-mon[33745]: from='client.48877 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForComplete_smithi016-74264-28"}]': finished 2024-03-26T04:21:36.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:36 smithi183 ceph-mon[33745]: from='client.48889 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiWritePP_smithi016-74285-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:36.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:36 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-15","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:36.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:36 smithi183 ceph-mon[33745]: from='client.48826 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21"}]': finished 2024-03-26T04:21:36.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:36 smithi183 ceph-mon[33745]: osdmap e172: 8 total, 8 up, 8 in 2024-03-26T04:21:36.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:36 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2588716529' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi016-74264-28"}]: dispatch 2024-03-26T04:21:36.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:36 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3873220630' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21"}]: dispatch 2024-03-26T04:21:36.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:36 smithi183 ceph-mon[33745]: from='client.48877 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi016-74264-28"}]: dispatch 2024-03-26T04:21:37.085 INFO:tasks.workunit.client.0.smithi016.stdout:hotsECPP.SnapRemovePP (2006 ms) 2024-03-26T04:21:37.085 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-03-26T04:21:37.085 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (2033 ms) 2024-03-26T04:21:37.085 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-03-26T04:21:37.085 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (1928 ms) 2024-03-26T04:21:37.085 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (8981 ms total) 2024-03-26T04:21:37.085 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: 2024-03-26T04:21:37.085 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-03-26T04:21:37.085 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-03-26T04:21:37.086 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (4037 ms) 2024-03-26T04:21:37.086 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-03-26T04:21:37.086 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (3982 ms) 2024-03-26T04:21:37.086 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-03-26T04:21:37.086 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (3965 ms) 2024-03-26T04:21:37.086 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (11984 ms total) 2024-03-26T04:21:37.086 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: 2024-03-26T04:21:37.086 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-03-26T04:21:37.086 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (108424 ms total) 2024-03-26T04:21:37.087 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-03-26T04:21:37.087 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-03-26T04:21:37.087 INFO:tasks.workunit.client.0.smithi016.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-03-26T04:21:37.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:37 smithi016 ceph-mon[37215]: from='client.48826 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21"}]: dispatch 2024-03-26T04:21:37.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:37 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:37.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:37 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:21:37.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:37 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:21:37.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:37 smithi016 ceph-mon[37215]: from='client.48877 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi016-74264-28"}]': finished 2024-03-26T04:21:37.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:37 smithi016 ceph-mon[37215]: from='client.48826 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21"}]': finished 2024-03-26T04:21:37.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:37 smithi016 ceph-mon[37215]: osdmap e173: 8 total, 8 up, 8 in 2024-03-26T04:21:37.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:37 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:37.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:37 smithi016 ceph-mon[29101]: from='client.48826 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21"}]: dispatch 2024-03-26T04:21:37.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:37 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:37.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:37 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:21:37.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:37 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:21:37.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:37 smithi016 ceph-mon[29101]: from='client.48877 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi016-74264-28"}]': finished 2024-03-26T04:21:37.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:37 smithi016 ceph-mon[29101]: from='client.48826 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21"}]': finished 2024-03-26T04:21:37.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:37 smithi016 ceph-mon[29101]: osdmap e173: 8 total, 8 up, 8 in 2024-03-26T04:21:37.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:37 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:37.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:37 smithi183 ceph-mon[33745]: from='client.48826 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21"}]: dispatch 2024-03-26T04:21:37.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:37 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:37.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:37 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:21:37.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:37 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:21:37.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:37 smithi183 ceph-mon[33745]: from='client.48877 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForComplete_smithi016-74264-28"}]': finished 2024-03-26T04:21:37.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:37 smithi183 ceph-mon[33745]: from='client.48826 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosSnapshotsSelfManagedECPP_smithi016-74609-21"}]': finished 2024-03-26T04:21:37.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:37 smithi183 ceph-mon[33745]: osdmap e173: 8 total, 8 up, 8 in 2024-03-26T04:21:37.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:37 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:38.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:38 smithi016 ceph-mon[29101]: pgmap v218: 356 pgs: 58 unknown, 298 active+clean; 585 KiB data, 575 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:21:38.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:38 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:21:38.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:38 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:21:38.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:38 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2242854476' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi016-74264-29"}]: dispatch 2024-03-26T04:21:38.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:38 smithi016 ceph-mon[29101]: from='client.48901 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi016-74264-29"}]: dispatch 2024-03-26T04:21:38.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:38 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2242854476' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi016-74264-29"}]: dispatch 2024-03-26T04:21:38.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:38 smithi016 ceph-mon[29101]: from='client.48901 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi016-74264-29"}]: dispatch 2024-03-26T04:21:38.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:38 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2242854476' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi016-74264-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:38.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:38 smithi016 ceph-mon[29101]: from='client.48901 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi016-74264-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:38.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:38 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:38.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:38 smithi016 ceph-mon[37215]: pgmap v218: 356 pgs: 58 unknown, 298 active+clean; 585 KiB data, 575 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:21:38.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:38 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:21:38.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:38 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:21:38.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:38 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2242854476' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi016-74264-29"}]: dispatch 2024-03-26T04:21:38.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:38 smithi016 ceph-mon[37215]: from='client.48901 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi016-74264-29"}]: dispatch 2024-03-26T04:21:38.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:38 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2242854476' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi016-74264-29"}]: dispatch 2024-03-26T04:21:38.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:38 smithi016 ceph-mon[37215]: from='client.48901 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi016-74264-29"}]: dispatch 2024-03-26T04:21:38.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:38 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2242854476' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi016-74264-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:38.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:38 smithi016 ceph-mon[37215]: from='client.48901 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi016-74264-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:38.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:38 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:38.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:38 smithi183 ceph-mon[33745]: pgmap v218: 356 pgs: 58 unknown, 298 active+clean; 585 KiB data, 575 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:21:38.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:38 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:21:38.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:38 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-15", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:21:38.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:38 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2242854476' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi016-74264-29"}]: dispatch 2024-03-26T04:21:38.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:38 smithi183 ceph-mon[33745]: from='client.48901 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi016-74264-29"}]: dispatch 2024-03-26T04:21:38.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:38 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2242854476' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi016-74264-29"}]: dispatch 2024-03-26T04:21:38.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:38 smithi183 ceph-mon[33745]: from='client.48901 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi016-74264-29"}]: dispatch 2024-03-26T04:21:38.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:38 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2242854476' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi016-74264-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:38.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:38 smithi183 ceph-mon[33745]: from='client.48901 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi016-74264-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:38.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:38 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:39.106 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:21:38 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:21:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:21:39.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:39 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-15", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:21:39.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:39 smithi016 ceph-mon[29101]: from='client.48901 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi016-74264-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:21:39.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:39 smithi016 ceph-mon[29101]: osdmap e174: 8 total, 8 up, 8 in 2024-03-26T04:21:39.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:39 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2732560326' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi016-74285-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:39.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:39 smithi016 ceph-mon[29101]: from='client.56369 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi016-74285-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:39.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:39 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-15"}]: dispatch 2024-03-26T04:21:39.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:39 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2242854476' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi016-74264-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi016-74264-29"}]: dispatch 2024-03-26T04:21:39.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:39 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-15"}]: dispatch 2024-03-26T04:21:39.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:39 smithi016 ceph-mon[29101]: from='client.48901 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi016-74264-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi016-74264-29"}]: dispatch 2024-03-26T04:21:39.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:39 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:39.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:39 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-15", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:21:39.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:39 smithi016 ceph-mon[37215]: from='client.48901 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi016-74264-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:21:39.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:39 smithi016 ceph-mon[37215]: osdmap e174: 8 total, 8 up, 8 in 2024-03-26T04:21:39.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:39 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2732560326' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi016-74285-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:39.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:39 smithi016 ceph-mon[37215]: from='client.56369 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi016-74285-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:39.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:39 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-15"}]: dispatch 2024-03-26T04:21:39.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:39 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2242854476' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi016-74264-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi016-74264-29"}]: dispatch 2024-03-26T04:21:39.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:39 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-15"}]: dispatch 2024-03-26T04:21:39.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:39 smithi016 ceph-mon[37215]: from='client.48901 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi016-74264-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi016-74264-29"}]: dispatch 2024-03-26T04:21:39.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:39 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:39.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:39 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-15", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:21:39.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:39 smithi183 ceph-mon[33745]: from='client.48901 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip_smithi016-74264-29", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:21:39.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:39 smithi183 ceph-mon[33745]: osdmap e174: 8 total, 8 up, 8 in 2024-03-26T04:21:39.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:39 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2732560326' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi016-74285-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:39.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:39 smithi183 ceph-mon[33745]: from='client.56369 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi016-74285-28","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:39.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:39 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-15"}]: dispatch 2024-03-26T04:21:39.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:39 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2242854476' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi016-74264-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi016-74264-29"}]: dispatch 2024-03-26T04:21:39.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:39 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-15"}]: dispatch 2024-03-26T04:21:39.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:39 smithi183 ceph-mon[33745]: from='client.48901 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip_smithi016-74264-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi016-74264-29"}]: dispatch 2024-03-26T04:21:39.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:39 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:40.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:40 smithi016 ceph-mon[29101]: pgmap v221: 356 pgs: 32 unknown, 324 active+clean; 585 KiB data, 583 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:21:40.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:40 smithi016 ceph-mon[29101]: from='client.56369 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi016-74285-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:40.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:40 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-15"}]': finished 2024-03-26T04:21:40.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:40 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-15", "mode": "writeback"}]: dispatch 2024-03-26T04:21:40.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:40 smithi016 ceph-mon[29101]: osdmap e175: 8 total, 8 up, 8 in 2024-03-26T04:21:40.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:40 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-15", "mode": "writeback"}]: dispatch 2024-03-26T04:21:40.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:40 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:40.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:40 smithi016 ceph-mon[37215]: pgmap v221: 356 pgs: 32 unknown, 324 active+clean; 585 KiB data, 583 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:21:40.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:40 smithi016 ceph-mon[37215]: from='client.56369 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi016-74285-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:40.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:40 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-15"}]': finished 2024-03-26T04:21:40.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:40 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-15", "mode": "writeback"}]: dispatch 2024-03-26T04:21:40.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:40 smithi016 ceph-mon[37215]: osdmap e175: 8 total, 8 up, 8 in 2024-03-26T04:21:40.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:40 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-15", "mode": "writeback"}]: dispatch 2024-03-26T04:21:40.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:40 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:40.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:40 smithi183 ceph-mon[33745]: pgmap v221: 356 pgs: 32 unknown, 324 active+clean; 585 KiB data, 583 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:21:40.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:40 smithi183 ceph-mon[33745]: from='client.56369 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "AioUnlockPP_smithi016-74285-28","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:40.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:40 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-15"}]': finished 2024-03-26T04:21:40.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:40 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-15", "mode": "writeback"}]: dispatch 2024-03-26T04:21:40.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:40 smithi183 ceph-mon[33745]: osdmap e175: 8 total, 8 up, 8 in 2024-03-26T04:21:40.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:40 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-15", "mode": "writeback"}]: dispatch 2024-03-26T04:21:40.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:40 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:41.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:41 smithi016 ceph-mon[29101]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:21:41.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:41 smithi016 ceph-mon[29101]: from='client.48901 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi016-74264-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi016-74264-29"}]': finished 2024-03-26T04:21:41.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:41 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-15", "mode": "writeback"}]': finished 2024-03-26T04:21:41.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:41 smithi016 ceph-mon[29101]: osdmap e176: 8 total, 8 up, 8 in 2024-03-26T04:21:41.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:41 smithi016 ceph-mon[29101]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:21:41.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:41 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:41.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:41 smithi016 ceph-mon[37215]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:21:41.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:41 smithi016 ceph-mon[37215]: from='client.48901 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi016-74264-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi016-74264-29"}]': finished 2024-03-26T04:21:41.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:41 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-15", "mode": "writeback"}]': finished 2024-03-26T04:21:41.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:41 smithi016 ceph-mon[37215]: osdmap e176: 8 total, 8 up, 8 in 2024-03-26T04:21:41.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:41 smithi016 ceph-mon[37215]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:21:41.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:41 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:41.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:41 smithi183 ceph-mon[33745]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:21:41.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:41 smithi183 ceph-mon[33745]: from='client.48901 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip_smithi016-74264-29", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip_smithi016-74264-29"}]': finished 2024-03-26T04:21:41.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:41 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-15", "mode": "writeback"}]': finished 2024-03-26T04:21:41.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:41 smithi183 ceph-mon[33745]: osdmap e176: 8 total, 8 up, 8 in 2024-03-26T04:21:41.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:41 smithi183 ceph-mon[33745]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:21:41.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:41 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:42.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:42 smithi016 ceph-mon[29101]: pgmap v224: 332 pgs: 8 unknown, 324 active+clean; 585 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:21:42.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:42 smithi016 ceph-mon[29101]: osdmap e177: 8 total, 8 up, 8 in 2024-03-26T04:21:42.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:42 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/4115660459' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi016-74285-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:42.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:42 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:21:42.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:42 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:21:42.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:42 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:42.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:42 smithi016 ceph-mon[37215]: pgmap v224: 332 pgs: 8 unknown, 324 active+clean; 585 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:21:42.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:42 smithi016 ceph-mon[37215]: osdmap e177: 8 total, 8 up, 8 in 2024-03-26T04:21:42.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:42 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/4115660459' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi016-74285-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:42.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:42 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:21:42.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:42 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:21:42.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:42 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:42.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:21:42 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:21:42.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:42 smithi183 ceph-mon[33745]: pgmap v224: 332 pgs: 8 unknown, 324 active+clean; 585 KiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:21:42.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:42 smithi183 ceph-mon[33745]: osdmap e177: 8 total, 8 up, 8 in 2024-03-26T04:21:42.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:42 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/4115660459' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi016-74285-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:42.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:42 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:21:42.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:42 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:21:42.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:42 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:43.139 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-03-26T04:21:43.139 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-03-26T04:21:43.139 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-03-26T04:21:43.139 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-03-26T04:21:43.139 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-03-26T04:21:43.139 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (11230 ms) 2024-03-26T04:21:43.139 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-03-26T04:21:43.140 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (23428 ms) 2024-03-26T04:21:43.140 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-03-26T04:21:43.140 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (6902 ms) 2024-03-26T04:21:43.140 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-03-26T04:21:43.140 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (4032 ms) 2024-03-26T04:21:43.140 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-03-26T04:21:43.140 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (2939 ms) 2024-03-26T04:21:43.140 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-03-26T04:21:43.140 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (3011 ms) 2024-03-26T04:21:43.140 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-03-26T04:21:43.140 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (2997 ms) 2024-03-26T04:21:43.140 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-03-26T04:21:43.140 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (3255 ms) 2024-03-26T04:21:43.140 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-03-26T04:21:43.141 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (2745 ms) 2024-03-26T04:21:43.141 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-03-26T04:21:43.141 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (2996 ms) 2024-03-26T04:21:43.141 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-03-26T04:21:43.141 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (3015 ms) 2024-03-26T04:21:43.141 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-03-26T04:21:43.141 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (2994 ms) 2024-03-26T04:21:43.141 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-03-26T04:21:43.141 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3004 ms) 2024-03-26T04:21:43.141 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-03-26T04:21:43.141 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (3005 ms) 2024-03-26T04:21:43.141 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-03-26T04:21:43.141 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3005 ms) 2024-03-26T04:21:43.141 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-03-26T04:21:43.142 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (3014 ms) 2024-03-26T04:21:43.142 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-03-26T04:21:43.142 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (3010 ms) 2024-03-26T04:21:43.142 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-03-26T04:21:43.142 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (2960 ms) 2024-03-26T04:21:43.142 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-03-26T04:21:43.142 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3011 ms) 2024-03-26T04:21:43.142 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-03-26T04:21:43.142 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (3015 ms) 2024-03-26T04:21:43.142 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-03-26T04:21:43.142 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (3274 ms) 2024-03-26T04:21:43.142 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-03-26T04:21:43.143 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (3003 ms) 2024-03-26T04:21:43.143 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-03-26T04:21:43.143 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (2701 ms) 2024-03-26T04:21:43.143 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-03-26T04:21:43.143 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (3005 ms) 2024-03-26T04:21:43.143 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-03-26T04:21:43.143 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (2997 ms) 2024-03-26T04:21:43.143 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-03-26T04:21:43.143 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3037 ms) 2024-03-26T04:21:43.143 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-03-26T04:21:43.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:43 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/4115660459' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi016-74285-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:43.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:43 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:21:43.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:43 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-15"}]: dispatch 2024-03-26T04:21:43.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:43 smithi016 ceph-mon[29101]: osdmap e178: 8 total, 8 up, 8 in 2024-03-26T04:21:43.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:43 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-15"}]: dispatch 2024-03-26T04:21:43.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:43 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2242854476' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi016-74264-29"}]: dispatch 2024-03-26T04:21:43.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:43 smithi016 ceph-mon[29101]: from='client.48901 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi016-74264-29"}]: dispatch 2024-03-26T04:21:43.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:43 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:21:43.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:43 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:43.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:43 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/4115660459' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi016-74285-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:43.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:43 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:21:43.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:43 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-15"}]: dispatch 2024-03-26T04:21:43.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:43 smithi016 ceph-mon[37215]: osdmap e178: 8 total, 8 up, 8 in 2024-03-26T04:21:43.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:43 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-15"}]: dispatch 2024-03-26T04:21:43.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:43 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2242854476' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi016-74264-29"}]: dispatch 2024-03-26T04:21:43.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:43 smithi016 ceph-mon[37215]: from='client.48901 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi016-74264-29"}]: dispatch 2024-03-26T04:21:43.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:43 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:21:43.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:43 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:43.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:43 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/4115660459' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripAppendPP_smithi016-74285-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:43.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:43 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:21:43.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:43 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-15"}]: dispatch 2024-03-26T04:21:43.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:43 smithi183 ceph-mon[33745]: osdmap e178: 8 total, 8 up, 8 in 2024-03-26T04:21:43.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:43 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-15"}]: dispatch 2024-03-26T04:21:43.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:43 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2242854476' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi016-74264-29"}]: dispatch 2024-03-26T04:21:43.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:43 smithi183 ceph-mon[33745]: from='client.48901 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi016-74264-29"}]: dispatch 2024-03-26T04:21:43.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:43 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:21:43.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:43 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:44.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:44 smithi016 ceph-mon[29101]: pgmap v227: 356 pgs: 18 creating+peering, 338 active+clean; 585 KiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:21:44.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:44 smithi016 ceph-mon[29101]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:21:44.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:44 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-15"}]': finished 2024-03-26T04:21:44.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:44 smithi016 ceph-mon[29101]: from='client.48901 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi016-74264-29"}]': finished 2024-03-26T04:21:44.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:44 smithi016 ceph-mon[29101]: osdmap e179: 8 total, 8 up, 8 in 2024-03-26T04:21:44.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:44 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2242854476' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi016-74264-29"}]: dispatch 2024-03-26T04:21:44.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:44 smithi016 ceph-mon[29101]: from='client.48901 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi016-74264-29"}]: dispatch 2024-03-26T04:21:44.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:44 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:44.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:44 smithi016 ceph-mon[37215]: pgmap v227: 356 pgs: 18 creating+peering, 338 active+clean; 585 KiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:21:44.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:44 smithi016 ceph-mon[37215]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:21:44.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:44 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-15"}]': finished 2024-03-26T04:21:44.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:44 smithi016 ceph-mon[37215]: from='client.48901 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi016-74264-29"}]': finished 2024-03-26T04:21:44.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:44 smithi016 ceph-mon[37215]: osdmap e179: 8 total, 8 up, 8 in 2024-03-26T04:21:44.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:44 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2242854476' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi016-74264-29"}]: dispatch 2024-03-26T04:21:44.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:44 smithi016 ceph-mon[37215]: from='client.48901 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi016-74264-29"}]: dispatch 2024-03-26T04:21:44.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:44 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:44.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:44 smithi183 ceph-mon[33745]: pgmap v227: 356 pgs: 18 creating+peering, 338 active+clean; 585 KiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:21:44.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:44 smithi183 ceph-mon[33745]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:21:44.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:44 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-15"}]': finished 2024-03-26T04:21:44.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:44 smithi183 ceph-mon[33745]: from='client.48901 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip_smithi016-74264-29"}]': finished 2024-03-26T04:21:44.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:44 smithi183 ceph-mon[33745]: osdmap e179: 8 total, 8 up, 8 in 2024-03-26T04:21:44.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:44 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2242854476' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi016-74264-29"}]: dispatch 2024-03-26T04:21:44.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:44 smithi183 ceph-mon[33745]: from='client.48901 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi016-74264-29"}]: dispatch 2024-03-26T04:21:44.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:44 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:45.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:45 smithi016 ceph-mon[37215]: from='client.48901 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi016-74264-29"}]': finished 2024-03-26T04:21:45.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:45 smithi016 ceph-mon[37215]: osdmap e180: 8 total, 8 up, 8 in 2024-03-26T04:21:45.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:45 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3402819135' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi016-74264-30"}]: dispatch 2024-03-26T04:21:45.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:45 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1094597652' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi016-74285-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:45.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:45 smithi016 ceph-mon[37215]: from='client.48919 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi016-74264-30"}]: dispatch 2024-03-26T04:21:45.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:45 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3402819135' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi016-74264-30"}]: dispatch 2024-03-26T04:21:45.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:45 smithi016 ceph-mon[37215]: from='client.48919 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi016-74264-30"}]: dispatch 2024-03-26T04:21:45.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:45 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3402819135' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi016-74264-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:45.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:45 smithi016 ceph-mon[37215]: from='client.48919 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi016-74264-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:45.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:45 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:45.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:45 smithi016 ceph-mon[29101]: from='client.48901 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi016-74264-29"}]': finished 2024-03-26T04:21:45.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:45 smithi016 ceph-mon[29101]: osdmap e180: 8 total, 8 up, 8 in 2024-03-26T04:21:45.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:45 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3402819135' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi016-74264-30"}]: dispatch 2024-03-26T04:21:45.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:45 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1094597652' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi016-74285-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:45.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:45 smithi016 ceph-mon[29101]: from='client.48919 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi016-74264-30"}]: dispatch 2024-03-26T04:21:45.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:45 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3402819135' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi016-74264-30"}]: dispatch 2024-03-26T04:21:45.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:45 smithi016 ceph-mon[29101]: from='client.48919 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi016-74264-30"}]: dispatch 2024-03-26T04:21:45.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:45 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3402819135' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi016-74264-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:45.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:45 smithi016 ceph-mon[29101]: from='client.48919 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi016-74264-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:45.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:45 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:45.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:45 smithi183 ceph-mon[33745]: from='client.48901 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip_smithi016-74264-29"}]': finished 2024-03-26T04:21:45.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:45 smithi183 ceph-mon[33745]: osdmap e180: 8 total, 8 up, 8 in 2024-03-26T04:21:45.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:45 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3402819135' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi016-74264-30"}]: dispatch 2024-03-26T04:21:45.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:45 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1094597652' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi016-74285-30","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:45.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:45 smithi183 ceph-mon[33745]: from='client.48919 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi016-74264-30"}]: dispatch 2024-03-26T04:21:45.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:45 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3402819135' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi016-74264-30"}]: dispatch 2024-03-26T04:21:45.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:45 smithi183 ceph-mon[33745]: from='client.48919 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi016-74264-30"}]: dispatch 2024-03-26T04:21:45.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:45 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3402819135' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi016-74264-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:45.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:45 smithi183 ceph-mon[33745]: from='client.48919 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi016-74264-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:45.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:45 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:46.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:46 smithi016 ceph-mon[29101]: pgmap v230: 324 pgs: 32 unknown, 292 active+clean; 585 KiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:21:46.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:46 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1094597652' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi016-74285-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:46.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:46 smithi016 ceph-mon[29101]: from='client.48919 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi016-74264-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:21:46.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:46 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3402819135' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi016-74264-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi016-74264-30"}]: dispatch 2024-03-26T04:21:46.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:46 smithi016 ceph-mon[29101]: osdmap e181: 8 total, 8 up, 8 in 2024-03-26T04:21:46.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:46 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:46.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:46 smithi016 ceph-mon[29101]: from='client.48919 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi016-74264-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi016-74264-30"}]: dispatch 2024-03-26T04:21:46.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:46 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:46.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:46 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:46.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:46 smithi016 ceph-mon[37215]: pgmap v230: 324 pgs: 32 unknown, 292 active+clean; 585 KiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:21:46.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:46 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1094597652' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi016-74285-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:46.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:46 smithi016 ceph-mon[37215]: from='client.48919 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi016-74264-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:21:46.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:46 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3402819135' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi016-74264-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi016-74264-30"}]: dispatch 2024-03-26T04:21:46.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:46 smithi016 ceph-mon[37215]: osdmap e181: 8 total, 8 up, 8 in 2024-03-26T04:21:46.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:46 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:46.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:46 smithi016 ceph-mon[37215]: from='client.48919 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi016-74264-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi016-74264-30"}]: dispatch 2024-03-26T04:21:46.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:46 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:46.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:46 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:46.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:46 smithi183 ceph-mon[33745]: pgmap v230: 324 pgs: 32 unknown, 292 active+clean; 585 KiB data, 588 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:21:46.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:46 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1094597652' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RacingRemovePP_smithi016-74285-30","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:46.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:46 smithi183 ceph-mon[33745]: from='client.48919 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTrip2_smithi016-74264-30", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:21:46.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:46 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3402819135' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi016-74264-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi016-74264-30"}]: dispatch 2024-03-26T04:21:46.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:46 smithi183 ceph-mon[33745]: osdmap e181: 8 total, 8 up, 8 in 2024-03-26T04:21:46.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:46 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:46.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:46 smithi183 ceph-mon[33745]: from='client.48919 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi016-74264-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi016-74264-30"}]: dispatch 2024-03-26T04:21:46.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:46 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-17","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:46.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:46 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-17","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:47.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:47 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:47.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:47 smithi016 ceph-mon[37215]: osdmap e182: 8 total, 8 up, 8 in 2024-03-26T04:21:47.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:47 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:21:47.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:47 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:21:47.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:47 smithi016 ceph-mon[37215]: from='client.48919 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi016-74264-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi016-74264-30"}]': finished 2024-03-26T04:21:47.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:47 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-17", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:21:47.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:47 smithi016 ceph-mon[37215]: osdmap e183: 8 total, 8 up, 8 in 2024-03-26T04:21:47.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:47 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1457839879' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi016-74285-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:47.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:47 smithi016 ceph-mon[37215]: from='client.56396 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi016-74285-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:47.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:47 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-17"}]: dispatch 2024-03-26T04:21:47.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:47 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-17"}]: dispatch 2024-03-26T04:21:47.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:47 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:47.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:47 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:47.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:47 smithi016 ceph-mon[29101]: osdmap e182: 8 total, 8 up, 8 in 2024-03-26T04:21:47.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:47 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:21:47.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:47 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:21:47.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:47 smithi016 ceph-mon[29101]: from='client.48919 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi016-74264-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi016-74264-30"}]': finished 2024-03-26T04:21:47.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:47 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-17", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:21:47.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:47 smithi016 ceph-mon[29101]: osdmap e183: 8 total, 8 up, 8 in 2024-03-26T04:21:47.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:47 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1457839879' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi016-74285-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:47.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:47 smithi016 ceph-mon[29101]: from='client.56396 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi016-74285-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:47.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:47 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-17"}]: dispatch 2024-03-26T04:21:47.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:47 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-17"}]: dispatch 2024-03-26T04:21:47.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:47 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:47.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:47 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:47.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:47 smithi183 ceph-mon[33745]: osdmap e182: 8 total, 8 up, 8 in 2024-03-26T04:21:47.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:47 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:21:47.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:47 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-17", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:21:47.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:47 smithi183 ceph-mon[33745]: from='client.48919 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTrip2_smithi016-74264-30", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTrip2_smithi016-74264-30"}]': finished 2024-03-26T04:21:47.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:47 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-17", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:21:47.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:47 smithi183 ceph-mon[33745]: osdmap e183: 8 total, 8 up, 8 in 2024-03-26T04:21:47.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:47 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1457839879' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi016-74285-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:47.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:47 smithi183 ceph-mon[33745]: from='client.56396 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi016-74285-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:47.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:47 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-17"}]: dispatch 2024-03-26T04:21:47.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:47 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-17"}]: dispatch 2024-03-26T04:21:47.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:47 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:48.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:48 smithi016 ceph-mon[37215]: pgmap v233: 324 pgs: 28 unknown, 296 active+clean; 585 KiB data, 588 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:21:48.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:48 smithi016 ceph-mon[37215]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:21:48.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:48 smithi016 ceph-mon[37215]: from='client.56396 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi016-74285-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:48.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:48 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-17"}]': finished 2024-03-26T04:21:48.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:48 smithi016 ceph-mon[37215]: osdmap e184: 8 total, 8 up, 8 in 2024-03-26T04:21:48.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:48 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-17", "mode": "writeback"}]: dispatch 2024-03-26T04:21:48.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:48 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-17", "mode": "writeback"}]: dispatch 2024-03-26T04:21:48.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:48 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:48.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:48 smithi016 ceph-mon[29101]: pgmap v233: 324 pgs: 28 unknown, 296 active+clean; 585 KiB data, 588 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:21:48.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:48 smithi016 ceph-mon[29101]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:21:48.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:48 smithi016 ceph-mon[29101]: from='client.56396 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi016-74285-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:48.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:48 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-17"}]': finished 2024-03-26T04:21:48.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:48 smithi016 ceph-mon[29101]: osdmap e184: 8 total, 8 up, 8 in 2024-03-26T04:21:48.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:48 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-17", "mode": "writeback"}]: dispatch 2024-03-26T04:21:48.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:48 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-17", "mode": "writeback"}]: dispatch 2024-03-26T04:21:48.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:48 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:48.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:48 smithi183 ceph-mon[33745]: pgmap v233: 324 pgs: 28 unknown, 296 active+clean; 585 KiB data, 588 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:21:48.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:48 smithi183 ceph-mon[33745]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:21:48.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:48 smithi183 ceph-mon[33745]: from='client.56396 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP_smithi016-74285-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:48.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:48 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-17"}]': finished 2024-03-26T04:21:48.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:48 smithi183 ceph-mon[33745]: osdmap e184: 8 total, 8 up, 8 in 2024-03-26T04:21:48.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:48 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-17", "mode": "writeback"}]: dispatch 2024-03-26T04:21:48.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:48 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-17", "mode": "writeback"}]: dispatch 2024-03-26T04:21:48.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:48 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:49.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:21:48 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:21:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:21:49.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:49 smithi016 ceph-mon[37215]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:21:49.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:49 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-17", "mode": "writeback"}]': finished 2024-03-26T04:21:49.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:49 smithi016 ceph-mon[37215]: osdmap e185: 8 total, 8 up, 8 in 2024-03-26T04:21:49.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:49 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3402819135' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi016-74264-30"}]: dispatch 2024-03-26T04:21:49.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:49 smithi016 ceph-mon[37215]: from='client.48919 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi016-74264-30"}]: dispatch 2024-03-26T04:21:49.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:49 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:49.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:49 smithi016 ceph-mon[29101]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:21:49.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:49 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-17", "mode": "writeback"}]': finished 2024-03-26T04:21:49.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:49 smithi016 ceph-mon[29101]: osdmap e185: 8 total, 8 up, 8 in 2024-03-26T04:21:49.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:49 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3402819135' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi016-74264-30"}]: dispatch 2024-03-26T04:21:49.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:49 smithi016 ceph-mon[29101]: from='client.48919 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi016-74264-30"}]: dispatch 2024-03-26T04:21:49.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:49 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:49.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:49 smithi183 ceph-mon[33745]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:21:49.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:49 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-17", "mode": "writeback"}]': finished 2024-03-26T04:21:49.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:49 smithi183 ceph-mon[33745]: osdmap e185: 8 total, 8 up, 8 in 2024-03-26T04:21:49.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:49 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3402819135' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi016-74264-30"}]: dispatch 2024-03-26T04:21:49.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:49 smithi183 ceph-mon[33745]: from='client.48919 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi016-74264-30"}]: dispatch 2024-03-26T04:21:49.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:49 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:50.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:50 smithi016 ceph-mon[37215]: pgmap v236: 364 pgs: 16 creating+peering, 24 unknown, 324 active+clean; 585 KiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:21:50.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:50 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:21:50.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:50 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:21:50.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:50 smithi016 ceph-mon[37215]: from='client.48919 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi016-74264-30"}]': finished 2024-03-26T04:21:50.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:50 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:21:50.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:50 smithi016 ceph-mon[37215]: osdmap e186: 8 total, 8 up, 8 in 2024-03-26T04:21:50.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:50 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3402819135' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi016-74264-30"}]: dispatch 2024-03-26T04:21:50.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:50 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3248906164' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi016-74285-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:50.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:50 smithi016 ceph-mon[37215]: from='client.56402 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi016-74285-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:50.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:50 smithi016 ceph-mon[37215]: from='client.48919 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi016-74264-30"}]: dispatch 2024-03-26T04:21:50.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:50 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-17"}]: dispatch 2024-03-26T04:21:50.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:50 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-17"}]: dispatch 2024-03-26T04:21:50.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:50 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:50.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:50 smithi016 ceph-mon[29101]: pgmap v236: 364 pgs: 16 creating+peering, 24 unknown, 324 active+clean; 585 KiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:21:50.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:50 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:21:50.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:50 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:21:50.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:50 smithi016 ceph-mon[29101]: from='client.48919 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi016-74264-30"}]': finished 2024-03-26T04:21:50.667 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:50 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:21:50.667 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:50 smithi016 ceph-mon[29101]: osdmap e186: 8 total, 8 up, 8 in 2024-03-26T04:21:50.667 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:50 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3402819135' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi016-74264-30"}]: dispatch 2024-03-26T04:21:50.667 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:50 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3248906164' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi016-74285-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:50.667 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:50 smithi016 ceph-mon[29101]: from='client.56402 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi016-74285-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:50.667 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:50 smithi016 ceph-mon[29101]: from='client.48919 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi016-74264-30"}]: dispatch 2024-03-26T04:21:50.668 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:50 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-17"}]: dispatch 2024-03-26T04:21:50.668 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:50 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-17"}]: dispatch 2024-03-26T04:21:50.668 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:50 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:50.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:50 smithi183 ceph-mon[33745]: pgmap v236: 364 pgs: 16 creating+peering, 24 unknown, 324 active+clean; 585 KiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:21:50.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:50 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:21:50.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:50 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:21:50.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:50 smithi183 ceph-mon[33745]: from='client.48919 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTrip2_smithi016-74264-30"}]': finished 2024-03-26T04:21:50.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:50 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:21:50.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:50 smithi183 ceph-mon[33745]: osdmap e186: 8 total, 8 up, 8 in 2024-03-26T04:21:50.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:50 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3402819135' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi016-74264-30"}]: dispatch 2024-03-26T04:21:50.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:50 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3248906164' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi016-74285-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:50.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:50 smithi183 ceph-mon[33745]: from='client.56402 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi016-74285-32","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:50.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:50 smithi183 ceph-mon[33745]: from='client.48919 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi016-74264-30"}]: dispatch 2024-03-26T04:21:50.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:50 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-17"}]: dispatch 2024-03-26T04:21:50.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:50 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-17"}]: dispatch 2024-03-26T04:21:50.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:50 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:51.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:51 smithi016 ceph-mon[29101]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:21:51.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:51 smithi016 ceph-mon[29101]: from='client.56402 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi016-74285-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:51.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:51 smithi016 ceph-mon[29101]: from='client.48919 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi016-74264-30"}]': finished 2024-03-26T04:21:51.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:51 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-17"}]': finished 2024-03-26T04:21:51.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:51 smithi016 ceph-mon[29101]: osdmap e187: 8 total, 8 up, 8 in 2024-03-26T04:21:51.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:51 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:51.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:51 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1286941543' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi016-74264-31"}]: dispatch 2024-03-26T04:21:51.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:51 smithi016 ceph-mon[29101]: from='client.56408 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi016-74264-31"}]: dispatch 2024-03-26T04:21:51.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:51 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1286941543' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi016-74264-31"}]: dispatch 2024-03-26T04:21:51.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:51 smithi016 ceph-mon[29101]: from='client.56408 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi016-74264-31"}]: dispatch 2024-03-26T04:21:51.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:51 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1286941543' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi016-74264-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:51.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:51 smithi016 ceph-mon[29101]: from='client.56408 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi016-74264-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:51.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:51 smithi016 ceph-mon[37215]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:21:51.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:51 smithi016 ceph-mon[37215]: from='client.56402 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi016-74285-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:51.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:51 smithi016 ceph-mon[37215]: from='client.48919 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi016-74264-30"}]': finished 2024-03-26T04:21:51.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:51 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-17"}]': finished 2024-03-26T04:21:51.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:51 smithi016 ceph-mon[37215]: osdmap e187: 8 total, 8 up, 8 in 2024-03-26T04:21:51.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:51 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:51.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:51 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1286941543' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi016-74264-31"}]: dispatch 2024-03-26T04:21:51.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:51 smithi016 ceph-mon[37215]: from='client.56408 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi016-74264-31"}]: dispatch 2024-03-26T04:21:51.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:51 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1286941543' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi016-74264-31"}]: dispatch 2024-03-26T04:21:51.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:51 smithi016 ceph-mon[37215]: from='client.56408 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi016-74264-31"}]: dispatch 2024-03-26T04:21:51.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:51 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1286941543' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi016-74264-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:51.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:51 smithi016 ceph-mon[37215]: from='client.56408 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi016-74264-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:51.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:51 smithi183 ceph-mon[33745]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:21:51.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:51 smithi183 ceph-mon[33745]: from='client.56402 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripCmpExtPP2_smithi016-74285-32","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:51.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:51 smithi183 ceph-mon[33745]: from='client.48919 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTrip2_smithi016-74264-30"}]': finished 2024-03-26T04:21:51.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:51 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-17"}]': finished 2024-03-26T04:21:51.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:51 smithi183 ceph-mon[33745]: osdmap e187: 8 total, 8 up, 8 in 2024-03-26T04:21:51.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:51 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:51.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:51 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1286941543' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi016-74264-31"}]: dispatch 2024-03-26T04:21:51.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:51 smithi183 ceph-mon[33745]: from='client.56408 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi016-74264-31"}]: dispatch 2024-03-26T04:21:51.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:51 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1286941543' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi016-74264-31"}]: dispatch 2024-03-26T04:21:51.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:51 smithi183 ceph-mon[33745]: from='client.56408 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi016-74264-31"}]: dispatch 2024-03-26T04:21:51.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:51 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1286941543' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi016-74264-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:51.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:51 smithi183 ceph-mon[33745]: from='client.56408 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi016-74264-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:52.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:21:52 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:21:52.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:52 smithi016 ceph-mon[37215]: pgmap v239: 356 pgs: 32 unknown, 324 active+clean; 585 KiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:21:52.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:52 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:21:52.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:52 smithi016 ceph-mon[37215]: from='client.56408 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi016-74264-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:21:52.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:52 smithi016 ceph-mon[37215]: osdmap e188: 8 total, 8 up, 8 in 2024-03-26T04:21:52.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:52 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1286941543' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi016-74264-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi016-74264-31"}]: dispatch 2024-03-26T04:21:52.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:52 smithi016 ceph-mon[37215]: from='client.56408 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi016-74264-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi016-74264-31"}]: dispatch 2024-03-26T04:21:52.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:52 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:52.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:52 smithi016 ceph-mon[29101]: pgmap v239: 356 pgs: 32 unknown, 324 active+clean; 585 KiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:21:52.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:52 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:21:52.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:52 smithi016 ceph-mon[29101]: from='client.56408 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi016-74264-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:21:52.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:52 smithi016 ceph-mon[29101]: osdmap e188: 8 total, 8 up, 8 in 2024-03-26T04:21:52.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:52 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1286941543' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi016-74264-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi016-74264-31"}]: dispatch 2024-03-26T04:21:52.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:52 smithi016 ceph-mon[29101]: from='client.56408 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi016-74264-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi016-74264-31"}]: dispatch 2024-03-26T04:21:52.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:52 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:53.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:52 smithi183 ceph-mon[33745]: pgmap v239: 356 pgs: 32 unknown, 324 active+clean; 585 KiB data, 589 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:21:53.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:52 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:21:53.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:52 smithi183 ceph-mon[33745]: from='client.56408 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppend_smithi016-74264-31", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:21:53.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:52 smithi183 ceph-mon[33745]: osdmap e188: 8 total, 8 up, 8 in 2024-03-26T04:21:53.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:52 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1286941543' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi016-74264-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi016-74264-31"}]: dispatch 2024-03-26T04:21:53.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:52 smithi183 ceph-mon[33745]: from='client.56408 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi016-74264-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi016-74264-31"}]: dispatch 2024-03-26T04:21:53.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:52 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:53.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:53 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:21:53.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:53 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi016-74374-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-26T04:21:53.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:53 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi016-74374-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-26T04:21:53.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:53 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi016-74374-1", "var": "pgp_num_actual", "val": "31"}]': finished 2024-03-26T04:21:53.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:53 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:53.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:53 smithi016 ceph-mon[37215]: osdmap e189: 8 total, 8 up, 8 in 2024-03-26T04:21:53.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:53 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1743761325' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi016-74285-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:53.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:53 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:53.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:53 smithi016 ceph-mon[37215]: from='client.56414 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi016-74285-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:53.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:53 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:53.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:53 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:21:53.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:53 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi016-74374-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-26T04:21:53.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:53 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi016-74374-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-26T04:21:53.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:53 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi016-74374-1", "var": "pgp_num_actual", "val": "31"}]': finished 2024-03-26T04:21:53.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:53 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:53.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:53 smithi016 ceph-mon[29101]: osdmap e189: 8 total, 8 up, 8 in 2024-03-26T04:21:53.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:53 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1743761325' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi016-74285-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:53.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:53 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:53.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:53 smithi016 ceph-mon[29101]: from='client.56414 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi016-74285-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:53.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:53 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:54.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:53 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:21:54.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:53 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi016-74374-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-26T04:21:54.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:53 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi016-74374-1", "var": "pgp_num_actual", "val": "31"}]: dispatch 2024-03-26T04:21:54.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:53 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi016-74374-1", "var": "pgp_num_actual", "val": "31"}]': finished 2024-03-26T04:21:54.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:53 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:54.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:53 smithi183 ceph-mon[33745]: osdmap e189: 8 total, 8 up, 8 in 2024-03-26T04:21:54.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:53 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1743761325' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi016-74285-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:54.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:53 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:54.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:53 smithi183 ceph-mon[33745]: from='client.56414 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi016-74285-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:54.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:53 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-19","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:54.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[37215]: pgmap v242: 292 pgs: 292 active+clean; 585 KiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:21:54.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[37215]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:21:54.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[37215]: from='client.56408 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi016-74264-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi016-74264-31"}]': finished 2024-03-26T04:21:54.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[37215]: from='client.56414 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi016-74285-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:54.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:54.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[37215]: osdmap e190: 8 total, 8 up, 8 in 2024-03-26T04:21:54.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:21:54.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:21:54.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:54.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:54.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi016-74374-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T04:21:54.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[37215]: from='client.24709 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi016-74374-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T04:21:54.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1743761325' entity='client.admin' cmd=[{ 2024-03-26T04:21:54.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[37215]: "prefix": "osd pool set", 2024-03-26T04:21:54.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[37215]: "pool": "PoolEIOFlag_smithi016-74285-33", 2024-03-26T04:21:54.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[37215]: "var": "eio", 2024-03-26T04:21:54.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[37215]: "val": "true" 2024-03-26T04:21:54.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[37215]: }]: dispatch 2024-03-26T04:21:54.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[37215]: from='client.56414 ' entity='client.admin' cmd=[{ 2024-03-26T04:21:54.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[37215]: "prefix": "osd pool set", 2024-03-26T04:21:54.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[37215]: "pool": "PoolEIOFlag_smithi016-74285-33", 2024-03-26T04:21:54.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[37215]: "var": "eio", 2024-03-26T04:21:54.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[37215]: "val": "true" 2024-03-26T04:21:54.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[37215]: }]: dispatch 2024-03-26T04:21:54.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[29101]: pgmap v242: 292 pgs: 292 active+clean; 585 KiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:21:54.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[29101]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:21:54.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[29101]: from='client.56408 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi016-74264-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi016-74264-31"}]': finished 2024-03-26T04:21:54.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[29101]: from='client.56414 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi016-74285-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:54.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:54.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[29101]: osdmap e190: 8 total, 8 up, 8 in 2024-03-26T04:21:54.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:21:54.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:21:54.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:54.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:54.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi016-74374-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T04:21:54.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[29101]: from='client.24709 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi016-74374-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T04:21:54.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1743761325' entity='client.admin' cmd=[{ 2024-03-26T04:21:54.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[29101]: "prefix": "osd pool set", 2024-03-26T04:21:54.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[29101]: "pool": "PoolEIOFlag_smithi016-74285-33", 2024-03-26T04:21:54.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[29101]: "var": "eio", 2024-03-26T04:21:54.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[29101]: "val": "true" 2024-03-26T04:21:54.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[29101]: }]: dispatch 2024-03-26T04:21:54.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[29101]: from='client.56414 ' entity='client.admin' cmd=[{ 2024-03-26T04:21:54.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[29101]: "prefix": "osd pool set", 2024-03-26T04:21:54.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[29101]: "pool": "PoolEIOFlag_smithi016-74285-33", 2024-03-26T04:21:54.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[29101]: "var": "eio", 2024-03-26T04:21:54.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[29101]: "val": "true" 2024-03-26T04:21:54.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:54 smithi016 ceph-mon[29101]: }]: dispatch 2024-03-26T04:21:55.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:54 smithi183 ceph-mon[33745]: pgmap v242: 292 pgs: 292 active+clean; 585 KiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:21:55.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:54 smithi183 ceph-mon[33745]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:21:55.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:54 smithi183 ceph-mon[33745]: from='client.56408 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppend_smithi016-74264-31", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppend_smithi016-74264-31"}]': finished 2024-03-26T04:21:55.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:54 smithi183 ceph-mon[33745]: from='client.56414 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolEIOFlag_smithi016-74285-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:55.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:54 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-19","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:55.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:54 smithi183 ceph-mon[33745]: osdmap e190: 8 total, 8 up, 8 in 2024-03-26T04:21:55.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:54 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:21:55.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:54 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-19", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:21:55.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:54 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:55.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:54 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:55.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:54 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi016-74374-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T04:21:55.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:54 smithi183 ceph-mon[33745]: from='client.24709 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi016-74374-1","var":"pg_num","val":"11"}]: dispatch 2024-03-26T04:21:55.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:54 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1743761325' entity='client.admin' cmd=[{ 2024-03-26T04:21:55.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:54 smithi183 ceph-mon[33745]: "prefix": "osd pool set", 2024-03-26T04:21:55.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:54 smithi183 ceph-mon[33745]: "pool": "PoolEIOFlag_smithi016-74285-33", 2024-03-26T04:21:55.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:54 smithi183 ceph-mon[33745]: "var": "eio", 2024-03-26T04:21:55.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:54 smithi183 ceph-mon[33745]: "val": "true" 2024-03-26T04:21:55.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:54 smithi183 ceph-mon[33745]: }]: dispatch 2024-03-26T04:21:55.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:54 smithi183 ceph-mon[33745]: from='client.56414 ' entity='client.admin' cmd=[{ 2024-03-26T04:21:55.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:54 smithi183 ceph-mon[33745]: "prefix": "osd pool set", 2024-03-26T04:21:55.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:54 smithi183 ceph-mon[33745]: "pool": "PoolEIOFlag_smithi016-74285-33", 2024-03-26T04:21:55.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:54 smithi183 ceph-mon[33745]: "var": "eio", 2024-03-26T04:21:55.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:54 smithi183 ceph-mon[33745]: "val": "true" 2024-03-26T04:21:55.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:54 smithi183 ceph-mon[33745]: }]: dispatch 2024-03-26T04:21:56.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:56 smithi016 ceph-mon[29101]: pgmap v245: 364 pgs: 72 unknown, 292 active+clean; 585 KiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:21:56.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:56 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-19", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:21:56.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:56 smithi016 ceph-mon[29101]: from='client.24709 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi016-74374-1","var":"pg_num","val":"11"}]': finished 2024-03-26T04:21:56.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:56 smithi016 ceph-mon[29101]: from='client.56414 ' entity='client.admin' cmd='[{ 2024-03-26T04:21:56.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:56 smithi016 ceph-mon[29101]: "prefix": "osd pool set", 2024-03-26T04:21:56.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:56 smithi016 ceph-mon[29101]: "pool": "PoolEIOFlag_smithi016-74285-33", 2024-03-26T04:21:56.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:56 smithi016 ceph-mon[29101]: "var": "eio", 2024-03-26T04:21:56.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:56 smithi016 ceph-mon[29101]: "val": "true" 2024-03-26T04:21:56.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:56 smithi016 ceph-mon[29101]: }]': finished 2024-03-26T04:21:56.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:56 smithi016 ceph-mon[29101]: osdmap e191: 8 total, 8 up, 8 in 2024-03-26T04:21:56.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:56 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-19"}]: dispatch 2024-03-26T04:21:56.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:56 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:56.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:56 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:56.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:56 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-19"}]: dispatch 2024-03-26T04:21:56.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:56 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi016-74374-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T04:21:56.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:56 smithi016 ceph-mon[29101]: from='client.24709 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi016-74374-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T04:21:56.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:56 smithi016 ceph-mon[37215]: pgmap v245: 364 pgs: 72 unknown, 292 active+clean; 585 KiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:21:56.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:56 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-19", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:21:56.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:56 smithi016 ceph-mon[37215]: from='client.24709 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi016-74374-1","var":"pg_num","val":"11"}]': finished 2024-03-26T04:21:56.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:56 smithi016 ceph-mon[37215]: from='client.56414 ' entity='client.admin' cmd='[{ 2024-03-26T04:21:56.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:56 smithi016 ceph-mon[37215]: "prefix": "osd pool set", 2024-03-26T04:21:56.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:56 smithi016 ceph-mon[37215]: "pool": "PoolEIOFlag_smithi016-74285-33", 2024-03-26T04:21:56.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:56 smithi016 ceph-mon[37215]: "var": "eio", 2024-03-26T04:21:56.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:56 smithi016 ceph-mon[37215]: "val": "true" 2024-03-26T04:21:56.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:56 smithi016 ceph-mon[37215]: }]': finished 2024-03-26T04:21:56.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:56 smithi016 ceph-mon[37215]: osdmap e191: 8 total, 8 up, 8 in 2024-03-26T04:21:56.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:56 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-19"}]: dispatch 2024-03-26T04:21:56.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:56 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:56.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:56 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:56.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:56 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-19"}]: dispatch 2024-03-26T04:21:56.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:56 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi016-74374-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T04:21:56.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:56 smithi016 ceph-mon[37215]: from='client.24709 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi016-74374-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T04:21:56.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:56 smithi183 ceph-mon[33745]: pgmap v245: 364 pgs: 72 unknown, 292 active+clean; 585 KiB data, 590 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:21:56.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:56 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-19", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:21:56.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:56 smithi183 ceph-mon[33745]: from='client.24709 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi016-74374-1","var":"pg_num","val":"11"}]': finished 2024-03-26T04:21:56.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:56 smithi183 ceph-mon[33745]: from='client.56414 ' entity='client.admin' cmd='[{ 2024-03-26T04:21:56.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:56 smithi183 ceph-mon[33745]: "prefix": "osd pool set", 2024-03-26T04:21:56.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:56 smithi183 ceph-mon[33745]: "pool": "PoolEIOFlag_smithi016-74285-33", 2024-03-26T04:21:56.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:56 smithi183 ceph-mon[33745]: "var": "eio", 2024-03-26T04:21:56.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:56 smithi183 ceph-mon[33745]: "val": "true" 2024-03-26T04:21:56.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:56 smithi183 ceph-mon[33745]: }]': finished 2024-03-26T04:21:56.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:56 smithi183 ceph-mon[33745]: osdmap e191: 8 total, 8 up, 8 in 2024-03-26T04:21:56.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:56 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-19"}]: dispatch 2024-03-26T04:21:56.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:56 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:56.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:56 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:56.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:56 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-19"}]: dispatch 2024-03-26T04:21:56.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:56 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi016-74374-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T04:21:56.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:56 smithi183 ceph-mon[33745]: from='client.24709 ' entity='client.admin' cmd=[{"prefix":"osd pool set","pool":"LibRadosList_smithi016-74374-1","var":"pgp_num","val":"11"}]: dispatch 2024-03-26T04:21:57.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:57 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-19"}]': finished 2024-03-26T04:21:57.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:57 smithi016 ceph-mon[29101]: from='client.24709 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi016-74374-1","var":"pgp_num","val":"11"}]': finished 2024-03-26T04:21:57.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:57 smithi016 ceph-mon[29101]: osdmap e192: 8 total, 8 up, 8 in 2024-03-26T04:21:57.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:57 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-19", "mode": "writeback"}]: dispatch 2024-03-26T04:21:57.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:57 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:57.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:57 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-19", "mode": "writeback"}]: dispatch 2024-03-26T04:21:57.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:57 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1286941543' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi016-74264-31"}]: dispatch 2024-03-26T04:21:57.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:57 smithi016 ceph-mon[29101]: from='client.56408 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi016-74264-31"}]: dispatch 2024-03-26T04:21:57.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:57 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-19"}]': finished 2024-03-26T04:21:57.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:57 smithi016 ceph-mon[37215]: from='client.24709 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi016-74374-1","var":"pgp_num","val":"11"}]': finished 2024-03-26T04:21:57.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:57 smithi016 ceph-mon[37215]: osdmap e192: 8 total, 8 up, 8 in 2024-03-26T04:21:57.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:57 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-19", "mode": "writeback"}]: dispatch 2024-03-26T04:21:57.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:57 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:57.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:57 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-19", "mode": "writeback"}]: dispatch 2024-03-26T04:21:57.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:57 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1286941543' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi016-74264-31"}]: dispatch 2024-03-26T04:21:57.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:57 smithi016 ceph-mon[37215]: from='client.56408 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi016-74264-31"}]: dispatch 2024-03-26T04:21:57.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:57 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-19"}]': finished 2024-03-26T04:21:57.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:57 smithi183 ceph-mon[33745]: from='client.24709 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi016-74374-1","var":"pgp_num","val":"11"}]': finished 2024-03-26T04:21:57.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:57 smithi183 ceph-mon[33745]: osdmap e192: 8 total, 8 up, 8 in 2024-03-26T04:21:57.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:57 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-19", "mode": "writeback"}]: dispatch 2024-03-26T04:21:57.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:57 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:57.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:57 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-19", "mode": "writeback"}]: dispatch 2024-03-26T04:21:57.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:57 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1286941543' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi016-74264-31"}]: dispatch 2024-03-26T04:21:57.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:57 smithi183 ceph-mon[33745]: from='client.56408 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi016-74264-31"}]: dispatch 2024-03-26T04:21:58.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:58 smithi016 ceph-mon[29101]: pgmap v248: 324 pgs: 27 unknown, 297 active+clean; 585 KiB data, 590 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:21:58.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:58 smithi016 ceph-mon[29101]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:21:58.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:58.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:58 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-19", "mode": "writeback"}]': finished 2024-03-26T04:21:58.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:58 smithi016 ceph-mon[29101]: from='client.56408 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi016-74264-31"}]': finished 2024-03-26T04:21:58.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:58 smithi016 ceph-mon[29101]: osdmap e193: 8 total, 8 up, 8 in 2024-03-26T04:21:58.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1286941543' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi016-74264-31"}]: dispatch 2024-03-26T04:21:58.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:58 smithi016 ceph-mon[29101]: from='client.56408 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi016-74264-31"}]: dispatch 2024-03-26T04:21:58.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1669876377' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi016-74285-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:58.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:58 smithi016 ceph-mon[29101]: from='client.56420 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi016-74285-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:58.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:21:58.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:58 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:21:58.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:58 smithi016 ceph-mon[37215]: pgmap v248: 324 pgs: 27 unknown, 297 active+clean; 585 KiB data, 590 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:21:58.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:58 smithi016 ceph-mon[37215]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:21:58.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:58.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:58 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-19", "mode": "writeback"}]': finished 2024-03-26T04:21:58.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:58 smithi016 ceph-mon[37215]: from='client.56408 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi016-74264-31"}]': finished 2024-03-26T04:21:58.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:58 smithi016 ceph-mon[37215]: osdmap e193: 8 total, 8 up, 8 in 2024-03-26T04:21:58.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1286941543' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi016-74264-31"}]: dispatch 2024-03-26T04:21:58.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:58 smithi016 ceph-mon[37215]: from='client.56408 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi016-74264-31"}]: dispatch 2024-03-26T04:21:58.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1669876377' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi016-74285-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:58.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:58 smithi016 ceph-mon[37215]: from='client.56420 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi016-74285-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:58.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:21:58.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:58 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:21:58.443 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:58 smithi183 ceph-mon[33745]: pgmap v248: 324 pgs: 27 unknown, 297 active+clean; 585 KiB data, 590 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:21:58.443 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:58 smithi183 ceph-mon[33745]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:21:58.443 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:58.443 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:58 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-19", "mode": "writeback"}]': finished 2024-03-26T04:21:58.443 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:58 smithi183 ceph-mon[33745]: from='client.56408 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppend_smithi016-74264-31"}]': finished 2024-03-26T04:21:58.443 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:58 smithi183 ceph-mon[33745]: osdmap e193: 8 total, 8 up, 8 in 2024-03-26T04:21:58.444 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1286941543' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi016-74264-31"}]: dispatch 2024-03-26T04:21:58.444 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:58 smithi183 ceph-mon[33745]: from='client.56408 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi016-74264-31"}]: dispatch 2024-03-26T04:21:58.444 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1669876377' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi016-74285-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:58.444 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:58 smithi183 ceph-mon[33745]: from='client.56420 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "MultiReads_smithi016-74285-34","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:21:58.444 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:21:58.444 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:58 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:21:59.112 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:21:58 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:21:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:21:59.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:59 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:59.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:59 smithi016 ceph-mon[29101]: from='client.56408 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi016-74264-31"}]': finished 2024-03-26T04:21:59.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:59 smithi016 ceph-mon[29101]: from='client.56420 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi016-74285-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:59.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:59 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:21:59.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:59 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-19"}]: dispatch 2024-03-26T04:21:59.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:59 smithi016 ceph-mon[29101]: osdmap e194: 8 total, 8 up, 8 in 2024-03-26T04:21:59.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:59 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-19"}]: dispatch 2024-03-26T04:21:59.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:59 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/225464454' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi016-74264-32"}]: dispatch 2024-03-26T04:21:59.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:59 smithi016 ceph-mon[29101]: from='client.56423 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi016-74264-32"}]: dispatch 2024-03-26T04:21:59.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:59 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/225464454' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi016-74264-32"}]: dispatch 2024-03-26T04:21:59.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:59 smithi016 ceph-mon[29101]: from='client.56423 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi016-74264-32"}]: dispatch 2024-03-26T04:21:59.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:59 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/225464454' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi016-74264-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:59.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:59 smithi016 ceph-mon[29101]: from='client.56423 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi016-74264-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:59.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:59 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi016-74374-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-26T04:21:59.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:21:59 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi016-74374-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-26T04:21:59.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:59 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:59.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:59 smithi016 ceph-mon[37215]: from='client.56408 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi016-74264-31"}]': finished 2024-03-26T04:21:59.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:59 smithi016 ceph-mon[37215]: from='client.56420 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi016-74285-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:59.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:59 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:21:59.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:59 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-19"}]: dispatch 2024-03-26T04:21:59.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:59 smithi016 ceph-mon[37215]: osdmap e194: 8 total, 8 up, 8 in 2024-03-26T04:21:59.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:59 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-19"}]: dispatch 2024-03-26T04:21:59.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:59 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/225464454' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi016-74264-32"}]: dispatch 2024-03-26T04:21:59.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:59 smithi016 ceph-mon[37215]: from='client.56423 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi016-74264-32"}]: dispatch 2024-03-26T04:21:59.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:59 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/225464454' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi016-74264-32"}]: dispatch 2024-03-26T04:21:59.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:59 smithi016 ceph-mon[37215]: from='client.56423 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi016-74264-32"}]: dispatch 2024-03-26T04:21:59.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:59 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/225464454' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi016-74264-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:59.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:59 smithi016 ceph-mon[37215]: from='client.56423 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi016-74264-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:59.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:59 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi016-74374-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-26T04:21:59.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:21:59 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi016-74374-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-26T04:21:59.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:59 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:21:59.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:59 smithi183 ceph-mon[33745]: from='client.56408 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppend_smithi016-74264-31"}]': finished 2024-03-26T04:21:59.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:59 smithi183 ceph-mon[33745]: from='client.56420 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "MultiReads_smithi016-74285-34","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:21:59.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:59 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:21:59.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:59 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-19"}]: dispatch 2024-03-26T04:21:59.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:59 smithi183 ceph-mon[33745]: osdmap e194: 8 total, 8 up, 8 in 2024-03-26T04:21:59.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:59 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-19"}]: dispatch 2024-03-26T04:21:59.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:59 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/225464454' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi016-74264-32"}]: dispatch 2024-03-26T04:21:59.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:59 smithi183 ceph-mon[33745]: from='client.56423 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi016-74264-32"}]: dispatch 2024-03-26T04:21:59.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:59 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/225464454' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi016-74264-32"}]: dispatch 2024-03-26T04:21:59.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:59 smithi183 ceph-mon[33745]: from='client.56423 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi016-74264-32"}]: dispatch 2024-03-26T04:21:59.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:59 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/225464454' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi016-74264-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:59.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:59 smithi183 ceph-mon[33745]: from='client.56423 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi016-74264-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:21:59.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:59 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi016-74374-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-26T04:21:59.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:21:59 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi016-74374-1", "var": "pg_num_actual", "val": "31"}]: dispatch 2024-03-26T04:22:00.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:00 smithi016 ceph-mon[37215]: pgmap v251: 356 pgs: 13 creating+peering, 19 unknown, 324 active+clean; 585 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-03-26T04:22:00.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:00 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:00.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:00 smithi016 ceph-mon[37215]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:22:00.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:00 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-19"}]': finished 2024-03-26T04:22:00.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:00 smithi016 ceph-mon[37215]: from='client.56423 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi016-74264-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:22:00.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:00 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi016-74374-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-03-26T04:22:00.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:00 smithi016 ceph-mon[37215]: osdmap e195: 8 total, 8 up, 8 in 2024-03-26T04:22:00.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:00 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/225464454' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi016-74264-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi016-74264-32"}]: dispatch 2024-03-26T04:22:00.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:00 smithi016 ceph-mon[37215]: from='client.56423 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi016-74264-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi016-74264-32"}]: dispatch 2024-03-26T04:22:00.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:00 smithi016 ceph-mon[29101]: pgmap v251: 356 pgs: 13 creating+peering, 19 unknown, 324 active+clean; 585 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-03-26T04:22:00.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:00 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:00.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:00 smithi016 ceph-mon[29101]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:22:00.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:00 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-19"}]': finished 2024-03-26T04:22:00.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:00 smithi016 ceph-mon[29101]: from='client.56423 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi016-74264-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:22:00.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:00 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi016-74374-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-03-26T04:22:00.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:00 smithi016 ceph-mon[29101]: osdmap e195: 8 total, 8 up, 8 in 2024-03-26T04:22:00.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:00 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/225464454' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi016-74264-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi016-74264-32"}]: dispatch 2024-03-26T04:22:00.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:00 smithi016 ceph-mon[29101]: from='client.56423 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi016-74264-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi016-74264-32"}]: dispatch 2024-03-26T04:22:00.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:00 smithi183 ceph-mon[33745]: pgmap v251: 356 pgs: 13 creating+peering, 19 unknown, 324 active+clean; 585 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-03-26T04:22:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:00 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:00 smithi183 ceph-mon[33745]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:22:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:00 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-19"}]': finished 2024-03-26T04:22:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:00 smithi183 ceph-mon[33745]: from='client.56423 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsComplete_smithi016-74264-32", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:22:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:00 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi016-74374-1", "var": "pg_num_actual", "val": "31"}]': finished 2024-03-26T04:22:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:00 smithi183 ceph-mon[33745]: osdmap e195: 8 total, 8 up, 8 in 2024-03-26T04:22:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:00 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/225464454' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi016-74264-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi016-74264-32"}]: dispatch 2024-03-26T04:22:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:00 smithi183 ceph-mon[33745]: from='client.56423 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsComplete_smithi016-74264-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi016-74264-32"}]: dispatch 2024-03-26T04:22:01.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:01 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:01.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:01 smithi016 ceph-mon[37215]: osdmap e196: 8 total, 8 up, 8 in 2024-03-26T04:22:01.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:01 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3407686285' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi016-74285-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:01.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:01 smithi016 ceph-mon[37215]: from='client.56429 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi016-74285-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:01.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:01 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:01.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:01 smithi016 ceph-mon[29101]: osdmap e196: 8 total, 8 up, 8 in 2024-03-26T04:22:01.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:01 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3407686285' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi016-74285-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:01.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:01 smithi016 ceph-mon[29101]: from='client.56429 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi016-74285-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:01.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:01 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:01.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:01 smithi183 ceph-mon[33745]: osdmap e196: 8 total, 8 up, 8 in 2024-03-26T04:22:01.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:01 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3407686285' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi016-74285-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:01.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:01 smithi183 ceph-mon[33745]: from='client.56429 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi016-74285-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:02.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:02 smithi016 ceph-mon[37215]: pgmap v254: 324 pgs: 32 unknown, 292 active+clean; 585 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-03-26T04:22:02.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:02 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:02.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:02 smithi016 ceph-mon[37215]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:22:02.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:02 smithi016 ceph-mon[37215]: from='client.56423 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi016-74264-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi016-74264-32"}]': finished 2024-03-26T04:22:02.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:02 smithi016 ceph-mon[37215]: from='client.56429 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi016-74285-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:02.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:02 smithi016 ceph-mon[37215]: osdmap e197: 8 total, 8 up, 8 in 2024-03-26T04:22:02.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:02 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:02.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:02 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:02.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:02 smithi016 ceph-mon[29101]: pgmap v254: 324 pgs: 32 unknown, 292 active+clean; 585 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-03-26T04:22:02.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:02 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:02.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:02 smithi016 ceph-mon[29101]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:22:02.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:02 smithi016 ceph-mon[29101]: from='client.56423 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi016-74264-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi016-74264-32"}]': finished 2024-03-26T04:22:02.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:02 smithi016 ceph-mon[29101]: from='client.56429 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi016-74285-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:02.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:02 smithi016 ceph-mon[29101]: osdmap e197: 8 total, 8 up, 8 in 2024-03-26T04:22:02.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:02 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:02.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:02 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:02.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:22:02 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:22:02.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:02 smithi183 ceph-mon[33745]: pgmap v254: 324 pgs: 32 unknown, 292 active+clean; 585 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.2 KiB/s wr, 9 op/s; 31 B/s, 0 objects/s recovering 2024-03-26T04:22:02.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:02 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:02.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:02 smithi183 ceph-mon[33745]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:22:02.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:02 smithi183 ceph-mon[33745]: from='client.56423 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsComplete_smithi016-74264-32", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsComplete_smithi016-74264-32"}]': finished 2024-03-26T04:22:02.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:02 smithi183 ceph-mon[33745]: from='client.56429 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "ReadIntoBufferlist_smithi016-74285-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:02.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:02 smithi183 ceph-mon[33745]: osdmap e197: 8 total, 8 up, 8 in 2024-03-26T04:22:02.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:02 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:02.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:02 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-21","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:03.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:03 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:03.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:03 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:03.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:03 smithi016 ceph-mon[37215]: osdmap e198: 8 total, 8 up, 8 in 2024-03-26T04:22:03.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:03 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:22:03.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:03 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:22:03.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:03 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:22:03.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:03 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:03.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:03 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:03.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:03 smithi016 ceph-mon[29101]: osdmap e198: 8 total, 8 up, 8 in 2024-03-26T04:22:03.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:03 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:22:03.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:03 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:22:03.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:03 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:22:03.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:03 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:03.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:03 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-21","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:03.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:03 smithi183 ceph-mon[33745]: osdmap e198: 8 total, 8 up, 8 in 2024-03-26T04:22:03.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:03 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:22:03.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:03 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:22:03.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:03 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-21", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:22:04.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:04 smithi016 ceph-mon[37215]: pgmap v257: 331 pgs: 9 creating+activating, 24 creating+peering, 1 peering, 297 active+clean; 585 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:22:04.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:04 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:04.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:04 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-21", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:22:04.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:04 smithi016 ceph-mon[37215]: osdmap e199: 8 total, 8 up, 8 in 2024-03-26T04:22:04.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:04 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-21"}]: dispatch 2024-03-26T04:22:04.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:04 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-21"}]: dispatch 2024-03-26T04:22:04.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:04 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/225464454' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi016-74264-32"}]: dispatch 2024-03-26T04:22:04.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:04 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/749807333' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi016-74285-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:04.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:04 smithi016 ceph-mon[37215]: from='client.56423 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi016-74264-32"}]: dispatch 2024-03-26T04:22:04.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:04 smithi016 ceph-mon[29101]: pgmap v257: 331 pgs: 9 creating+activating, 24 creating+peering, 1 peering, 297 active+clean; 585 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:22:04.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:04 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:04.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:04 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-21", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:22:04.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:04 smithi016 ceph-mon[29101]: osdmap e199: 8 total, 8 up, 8 in 2024-03-26T04:22:04.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:04 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-21"}]: dispatch 2024-03-26T04:22:04.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:04 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-21"}]: dispatch 2024-03-26T04:22:04.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:04 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/225464454' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi016-74264-32"}]: dispatch 2024-03-26T04:22:04.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:04 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/749807333' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi016-74285-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:04.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:04 smithi016 ceph-mon[29101]: from='client.56423 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi016-74264-32"}]: dispatch 2024-03-26T04:22:04.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:04 smithi183 ceph-mon[33745]: pgmap v257: 331 pgs: 9 creating+activating, 24 creating+peering, 1 peering, 297 active+clean; 585 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:22:04.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:04 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:04.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:04 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-21", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:22:04.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:04 smithi183 ceph-mon[33745]: osdmap e199: 8 total, 8 up, 8 in 2024-03-26T04:22:04.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:04 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-21"}]: dispatch 2024-03-26T04:22:04.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:04 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-21"}]: dispatch 2024-03-26T04:22:04.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:04 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/225464454' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi016-74264-32"}]: dispatch 2024-03-26T04:22:04.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:04 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/749807333' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi016-74285-36","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:04.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:04 smithi183 ceph-mon[33745]: from='client.56423 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi016-74264-32"}]: dispatch 2024-03-26T04:22:05.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:05 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:05.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:05 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-21"}]': finished 2024-03-26T04:22:05.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:05 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/749807333' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi016-74285-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:05.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:05 smithi016 ceph-mon[37215]: from='client.56423 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi016-74264-32"}]': finished 2024-03-26T04:22:05.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:05 smithi016 ceph-mon[37215]: osdmap e200: 8 total, 8 up, 8 in 2024-03-26T04:22:05.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:05 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/225464454' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi016-74264-32"}]: dispatch 2024-03-26T04:22:05.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:05 smithi016 ceph-mon[37215]: from='client.56423 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi016-74264-32"}]: dispatch 2024-03-26T04:22:05.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:05 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-21", "mode": "writeback"}]: dispatch 2024-03-26T04:22:05.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:05 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-21", "mode": "writeback"}]: dispatch 2024-03-26T04:22:05.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:05 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:05.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:05 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-21"}]': finished 2024-03-26T04:22:05.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:05 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/749807333' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi016-74285-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:05.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:05 smithi016 ceph-mon[29101]: from='client.56423 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi016-74264-32"}]': finished 2024-03-26T04:22:05.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:05 smithi016 ceph-mon[29101]: osdmap e200: 8 total, 8 up, 8 in 2024-03-26T04:22:05.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:05 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/225464454' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi016-74264-32"}]: dispatch 2024-03-26T04:22:05.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:05 smithi016 ceph-mon[29101]: from='client.56423 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi016-74264-32"}]: dispatch 2024-03-26T04:22:05.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:05 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-21", "mode": "writeback"}]: dispatch 2024-03-26T04:22:05.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:05 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-21", "mode": "writeback"}]: dispatch 2024-03-26T04:22:05.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:05 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:05.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:05 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-21"}]': finished 2024-03-26T04:22:05.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:05 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/749807333' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi016-74285-36","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:05.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:05 smithi183 ceph-mon[33745]: from='client.56423 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsComplete_smithi016-74264-32"}]': finished 2024-03-26T04:22:05.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:05 smithi183 ceph-mon[33745]: osdmap e200: 8 total, 8 up, 8 in 2024-03-26T04:22:05.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:05 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/225464454' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi016-74264-32"}]: dispatch 2024-03-26T04:22:05.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:05 smithi183 ceph-mon[33745]: from='client.56423 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi016-74264-32"}]: dispatch 2024-03-26T04:22:05.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:05 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-21", "mode": "writeback"}]: dispatch 2024-03-26T04:22:05.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:05 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-21", "mode": "writeback"}]: dispatch 2024-03-26T04:22:06.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:06 smithi016 ceph-mon[29101]: pgmap v260: 355 pgs: 32 unknown, 6 creating+activating, 21 creating+peering, 1 peering, 295 active+clean; 585 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:22:06.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:06 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:06.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:06 smithi016 ceph-mon[29101]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:22:06.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:06 smithi016 ceph-mon[29101]: from='client.56423 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi016-74264-32"}]': finished 2024-03-26T04:22:06.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:06 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-21", "mode": "writeback"}]': finished 2024-03-26T04:22:06.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:06 smithi016 ceph-mon[29101]: osdmap e201: 8 total, 8 up, 8 in 2024-03-26T04:22:06.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:06 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3757336282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi016-74285-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:06.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:06 smithi016 ceph-mon[29101]: from='client.56441 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi016-74285-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:06.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:06 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2052626694' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi016-74264-33"}]: dispatch 2024-03-26T04:22:06.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:06 smithi016 ceph-mon[29101]: from='client.56447 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi016-74264-33"}]: dispatch 2024-03-26T04:22:06.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:06 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2052626694' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi016-74264-33"}]: dispatch 2024-03-26T04:22:06.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:06 smithi016 ceph-mon[29101]: from='client.56447 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi016-74264-33"}]: dispatch 2024-03-26T04:22:06.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:06 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2052626694' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi016-74264-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:06.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:06 smithi016 ceph-mon[29101]: from='client.56447 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi016-74264-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:06.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:06 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:22:06.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:06 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:22:06.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:06 smithi016 ceph-mon[37215]: pgmap v260: 355 pgs: 32 unknown, 6 creating+activating, 21 creating+peering, 1 peering, 295 active+clean; 585 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:22:06.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:06 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:06.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:06 smithi016 ceph-mon[37215]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:22:06.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:06 smithi016 ceph-mon[37215]: from='client.56423 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi016-74264-32"}]': finished 2024-03-26T04:22:06.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:06 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-21", "mode": "writeback"}]': finished 2024-03-26T04:22:06.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:06 smithi016 ceph-mon[37215]: osdmap e201: 8 total, 8 up, 8 in 2024-03-26T04:22:06.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:06 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3757336282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi016-74285-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:06.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:06 smithi016 ceph-mon[37215]: from='client.56441 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi016-74285-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:06.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:06 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2052626694' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi016-74264-33"}]: dispatch 2024-03-26T04:22:06.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:06 smithi016 ceph-mon[37215]: from='client.56447 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi016-74264-33"}]: dispatch 2024-03-26T04:22:06.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:06 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2052626694' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi016-74264-33"}]: dispatch 2024-03-26T04:22:06.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:06 smithi016 ceph-mon[37215]: from='client.56447 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi016-74264-33"}]: dispatch 2024-03-26T04:22:06.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:06 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2052626694' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi016-74264-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:06.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:06 smithi016 ceph-mon[37215]: from='client.56447 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi016-74264-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:06.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:06 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:22:06.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:06 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:22:06.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:06 smithi183 ceph-mon[33745]: pgmap v260: 355 pgs: 32 unknown, 6 creating+activating, 21 creating+peering, 1 peering, 295 active+clean; 585 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:22:06.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:06 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:06.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:06 smithi183 ceph-mon[33745]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:22:06.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:06 smithi183 ceph-mon[33745]: from='client.56423 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsComplete_smithi016-74264-32"}]': finished 2024-03-26T04:22:06.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:06 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-21", "mode": "writeback"}]': finished 2024-03-26T04:22:06.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:06 smithi183 ceph-mon[33745]: osdmap e201: 8 total, 8 up, 8 in 2024-03-26T04:22:06.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:06 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3757336282' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi016-74285-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:06.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:06 smithi183 ceph-mon[33745]: from='client.56441 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi016-74285-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:06.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:06 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2052626694' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi016-74264-33"}]: dispatch 2024-03-26T04:22:06.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:06 smithi183 ceph-mon[33745]: from='client.56447 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi016-74264-33"}]: dispatch 2024-03-26T04:22:06.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:06 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2052626694' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi016-74264-33"}]: dispatch 2024-03-26T04:22:06.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:06 smithi183 ceph-mon[33745]: from='client.56447 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi016-74264-33"}]: dispatch 2024-03-26T04:22:06.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:06 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2052626694' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi016-74264-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:06.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:06 smithi183 ceph-mon[33745]: from='client.56447 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi016-74264-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:06.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:06 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:22:06.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:06 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:22:07.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:07 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:07.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:07 smithi016 ceph-mon[37215]: from='client.56441 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi016-74285-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:07.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:07 smithi016 ceph-mon[37215]: from='client.56447 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi016-74264-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:22:07.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:07 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:22:07.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:07 smithi016 ceph-mon[37215]: osdmap e202: 8 total, 8 up, 8 in 2024-03-26T04:22:07.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:07 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-21"}]: dispatch 2024-03-26T04:22:07.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:07 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2052626694' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi016-74264-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi016-74264-33"}]: dispatch 2024-03-26T04:22:07.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:07 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-21"}]: dispatch 2024-03-26T04:22:07.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:07 smithi016 ceph-mon[37215]: from='client.56447 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi016-74264-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi016-74264-33"}]: dispatch 2024-03-26T04:22:07.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:07 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:22:07.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:07 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:22:07.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:07 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:07.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:07 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:07.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:07 smithi016 ceph-mon[29101]: from='client.56441 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi016-74285-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:07.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:07 smithi016 ceph-mon[29101]: from='client.56447 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi016-74264-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:22:07.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:07 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:22:07.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:07 smithi016 ceph-mon[29101]: osdmap e202: 8 total, 8 up, 8 in 2024-03-26T04:22:07.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:07 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-21"}]: dispatch 2024-03-26T04:22:07.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:07 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2052626694' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi016-74264-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi016-74264-33"}]: dispatch 2024-03-26T04:22:07.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:07 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-21"}]: dispatch 2024-03-26T04:22:07.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:07 smithi016 ceph-mon[29101]: from='client.56447 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi016-74264-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi016-74264-33"}]: dispatch 2024-03-26T04:22:07.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:07 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:22:07.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:07 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:22:07.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:07 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:07.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:07 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:07.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:07 smithi183 ceph-mon[33745]: from='client.56441 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi016-74285-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:07.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:07 smithi183 ceph-mon[33745]: from='client.56447 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafe_smithi016-74264-33", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:22:07.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:07 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:22:07.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:07 smithi183 ceph-mon[33745]: osdmap e202: 8 total, 8 up, 8 in 2024-03-26T04:22:07.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:07 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-21"}]: dispatch 2024-03-26T04:22:07.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:07 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2052626694' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi016-74264-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi016-74264-33"}]: dispatch 2024-03-26T04:22:07.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:07 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-21"}]: dispatch 2024-03-26T04:22:07.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:07 smithi183 ceph-mon[33745]: from='client.56447 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafe_smithi016-74264-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi016-74264-33"}]: dispatch 2024-03-26T04:22:07.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:07 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:22:07.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:07 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:22:07.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:07 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:08.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:08 smithi016 ceph-mon[37215]: pgmap v263: 387 pgs: 64 unknown, 6 creating+activating, 15 creating+peering, 1 peering, 301 active+clean; 585 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 511 B/s wr, 0 op/s 2024-03-26T04:22:08.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:08 smithi016 ceph-mon[37215]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:22:08.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:08 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-21"}]': finished 2024-03-26T04:22:08.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:08 smithi016 ceph-mon[37215]: osdmap e203: 8 total, 8 up, 8 in 2024-03-26T04:22:08.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:08 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1682219354' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi016-74285-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:08.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:08 smithi016 ceph-mon[37215]: from='client.56453 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi016-74285-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:08.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:08 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:08.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:08 smithi016 ceph-mon[29101]: pgmap v263: 387 pgs: 64 unknown, 6 creating+activating, 15 creating+peering, 1 peering, 301 active+clean; 585 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 511 B/s wr, 0 op/s 2024-03-26T04:22:08.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:08 smithi016 ceph-mon[29101]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:22:08.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:08 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-21"}]': finished 2024-03-26T04:22:08.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:08 smithi016 ceph-mon[29101]: osdmap e203: 8 total, 8 up, 8 in 2024-03-26T04:22:08.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:08 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1682219354' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi016-74285-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:08.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:08 smithi016 ceph-mon[29101]: from='client.56453 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi016-74285-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:08.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:08 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:08.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:08 smithi183 ceph-mon[33745]: pgmap v263: 387 pgs: 64 unknown, 6 creating+activating, 15 creating+peering, 1 peering, 301 active+clean; 585 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 511 B/s wr, 0 op/s 2024-03-26T04:22:08.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:08 smithi183 ceph-mon[33745]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:22:08.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:08 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-21"}]': finished 2024-03-26T04:22:08.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:08 smithi183 ceph-mon[33745]: osdmap e203: 8 total, 8 up, 8 in 2024-03-26T04:22:08.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:08 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1682219354' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi016-74285-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:08.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:08 smithi183 ceph-mon[33745]: from='client.56453 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi016-74285-38","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:08.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:08 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:09.155 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:22:08 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:22:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:22:09.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:09 smithi016 ceph-mon[37215]: from='client.56447 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi016-74264-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi016-74264-33"}]': finished 2024-03-26T04:22:09.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:09 smithi016 ceph-mon[37215]: from='client.56453 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi016-74285-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:09.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:09 smithi016 ceph-mon[37215]: osdmap e204: 8 total, 8 up, 8 in 2024-03-26T04:22:09.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:09 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:09.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:09 smithi016 ceph-mon[29101]: from='client.56447 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi016-74264-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi016-74264-33"}]': finished 2024-03-26T04:22:09.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:09 smithi016 ceph-mon[29101]: from='client.56453 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi016-74285-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:09.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:09 smithi016 ceph-mon[29101]: osdmap e204: 8 total, 8 up, 8 in 2024-03-26T04:22:09.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:09 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:09.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:09 smithi183 ceph-mon[33745]: from='client.56447 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafe_smithi016-74264-33", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafe_smithi016-74264-33"}]': finished 2024-03-26T04:22:09.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:09 smithi183 ceph-mon[33745]: from='client.56453 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrsRoundTripPP_smithi016-74285-38","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:09.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:09 smithi183 ceph-mon[33745]: osdmap e204: 8 total, 8 up, 8 in 2024-03-26T04:22:09.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:09 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:10.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:10 smithi016 ceph-mon[37215]: pgmap v266: 395 pgs: 40 unknown, 355 active+clean; 586 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:22:10.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:10 smithi016 ceph-mon[37215]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:22:10.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:10 smithi016 ceph-mon[37215]: osdmap e205: 8 total, 8 up, 8 in 2024-03-26T04:22:10.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:10 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:10.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:10 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:10.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:10 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:10.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:10 smithi016 ceph-mon[29101]: pgmap v266: 395 pgs: 40 unknown, 355 active+clean; 586 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:22:10.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:10 smithi016 ceph-mon[29101]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:22:10.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:10 smithi016 ceph-mon[29101]: osdmap e205: 8 total, 8 up, 8 in 2024-03-26T04:22:10.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:10 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:10.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:10 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:10.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:10 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:10.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:10 smithi183 ceph-mon[33745]: pgmap v266: 395 pgs: 40 unknown, 355 active+clean; 586 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:22:10.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:10 smithi183 ceph-mon[33745]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:22:10.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:10 smithi183 ceph-mon[33745]: osdmap e205: 8 total, 8 up, 8 in 2024-03-26T04:22:10.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:10 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:10.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:10 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-23","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:10.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:10 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:11.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:11 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:11.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:11 smithi183 ceph-mon[33745]: osdmap e206: 8 total, 8 up, 8 in 2024-03-26T04:22:11.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:11 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2052626694' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi016-74264-33"}]: dispatch 2024-03-26T04:22:11.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:11 smithi183 ceph-mon[33745]: from='client.56447 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi016-74264-33"}]: dispatch 2024-03-26T04:22:11.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:11 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:11.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:11 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:11.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:11 smithi016 ceph-mon[37215]: osdmap e206: 8 total, 8 up, 8 in 2024-03-26T04:22:11.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:11 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2052626694' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi016-74264-33"}]: dispatch 2024-03-26T04:22:11.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:11 smithi016 ceph-mon[37215]: from='client.56447 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi016-74264-33"}]: dispatch 2024-03-26T04:22:11.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:11 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:11.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:11 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-23","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:11.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:11 smithi016 ceph-mon[29101]: osdmap e206: 8 total, 8 up, 8 in 2024-03-26T04:22:11.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:11 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2052626694' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi016-74264-33"}]: dispatch 2024-03-26T04:22:11.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:11 smithi016 ceph-mon[29101]: from='client.56447 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi016-74264-33"}]: dispatch 2024-03-26T04:22:11.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:11 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:12.542 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:22:12 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:22:12.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:12 smithi183 ceph-mon[33745]: pgmap v269: 355 pgs: 32 unknown, 323 active+clean; 586 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:22:12.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:12 smithi183 ceph-mon[33745]: from='client.56447 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi016-74264-33"}]': finished 2024-03-26T04:22:12.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:12 smithi183 ceph-mon[33745]: osdmap e207: 8 total, 8 up, 8 in 2024-03-26T04:22:12.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:12 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2052626694' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi016-74264-33"}]: dispatch 2024-03-26T04:22:12.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:12 smithi183 ceph-mon[33745]: from='client.56447 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi016-74264-33"}]: dispatch 2024-03-26T04:22:12.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:12 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:22:12.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:12 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:22:12.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:12 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:12.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:12 smithi016 ceph-mon[37215]: pgmap v269: 355 pgs: 32 unknown, 323 active+clean; 586 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:22:12.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:12 smithi016 ceph-mon[37215]: from='client.56447 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi016-74264-33"}]': finished 2024-03-26T04:22:12.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:12 smithi016 ceph-mon[37215]: osdmap e207: 8 total, 8 up, 8 in 2024-03-26T04:22:12.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:12 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2052626694' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi016-74264-33"}]: dispatch 2024-03-26T04:22:12.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:12 smithi016 ceph-mon[37215]: from='client.56447 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi016-74264-33"}]: dispatch 2024-03-26T04:22:12.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:12 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:22:12.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:12 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:22:12.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:12 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:12.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:12 smithi016 ceph-mon[29101]: pgmap v269: 355 pgs: 32 unknown, 323 active+clean; 586 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:22:12.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:12 smithi016 ceph-mon[29101]: from='client.56447 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafe_smithi016-74264-33"}]': finished 2024-03-26T04:22:12.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:12 smithi016 ceph-mon[29101]: osdmap e207: 8 total, 8 up, 8 in 2024-03-26T04:22:12.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:12 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2052626694' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi016-74264-33"}]: dispatch 2024-03-26T04:22:12.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:12 smithi016 ceph-mon[29101]: from='client.56447 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi016-74264-33"}]: dispatch 2024-03-26T04:22:12.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:12 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:22:12.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:12 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-23", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:22:12.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:12 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:13.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:13 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:22:13.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:13 smithi183 ceph-mon[33745]: from='client.56447 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi016-74264-33"}]': finished 2024-03-26T04:22:13.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:13 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-23", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:22:13.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:13 smithi183 ceph-mon[33745]: osdmap e208: 8 total, 8 up, 8 in 2024-03-26T04:22:13.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:13 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-23"}]: dispatch 2024-03-26T04:22:13.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:13 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3102021829' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi016-74285-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:13.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:13 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-23"}]: dispatch 2024-03-26T04:22:13.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:13 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1477746817' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi016-74264-34"}]: dispatch 2024-03-26T04:22:13.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:13 smithi183 ceph-mon[33745]: from='client.56465 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi016-74264-34"}]: dispatch 2024-03-26T04:22:13.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:13 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1477746817' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi016-74264-34"}]: dispatch 2024-03-26T04:22:13.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:13 smithi183 ceph-mon[33745]: from='client.56465 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi016-74264-34"}]: dispatch 2024-03-26T04:22:13.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:13 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1477746817' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi016-74264-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:13.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:13 smithi183 ceph-mon[33745]: from='client.56465 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi016-74264-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:13.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:13 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:13.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:13 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3102021829' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi016-74285-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:13.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:13 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-23"}]': finished 2024-03-26T04:22:13.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:13 smithi183 ceph-mon[33745]: from='client.56465 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi016-74264-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:22:13.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:13 smithi183 ceph-mon[33745]: osdmap e209: 8 total, 8 up, 8 in 2024-03-26T04:22:13.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:13 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-23", "mode": "writeback"}]: dispatch 2024-03-26T04:22:13.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:13 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1477746817' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi016-74264-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi016-74264-34"}]: dispatch 2024-03-26T04:22:13.544 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:13 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-23", "mode": "writeback"}]: dispatch 2024-03-26T04:22:13.544 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:13 smithi183 ceph-mon[33745]: from='client.56465 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi016-74264-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi016-74264-34"}]: dispatch 2024-03-26T04:22:13.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:13 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:22:13.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:13 smithi016 ceph-mon[29101]: from='client.56447 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi016-74264-33"}]': finished 2024-03-26T04:22:13.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:13 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-23", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:22:13.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:13 smithi016 ceph-mon[29101]: osdmap e208: 8 total, 8 up, 8 in 2024-03-26T04:22:13.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:13 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-23"}]: dispatch 2024-03-26T04:22:13.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:13 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3102021829' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi016-74285-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:13.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:13 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-23"}]: dispatch 2024-03-26T04:22:13.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:13 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1477746817' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi016-74264-34"}]: dispatch 2024-03-26T04:22:13.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:13 smithi016 ceph-mon[29101]: from='client.56465 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi016-74264-34"}]: dispatch 2024-03-26T04:22:13.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:13 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1477746817' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi016-74264-34"}]: dispatch 2024-03-26T04:22:13.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:13 smithi016 ceph-mon[29101]: from='client.56465 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi016-74264-34"}]: dispatch 2024-03-26T04:22:13.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:13 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1477746817' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi016-74264-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:13.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:13 smithi016 ceph-mon[29101]: from='client.56465 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi016-74264-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:13.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:13 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:13.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:13 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3102021829' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi016-74285-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:13.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:13 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-23"}]': finished 2024-03-26T04:22:13.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:13 smithi016 ceph-mon[29101]: from='client.56465 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi016-74264-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:22:13.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:13 smithi016 ceph-mon[29101]: osdmap e209: 8 total, 8 up, 8 in 2024-03-26T04:22:13.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:13 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-23", "mode": "writeback"}]: dispatch 2024-03-26T04:22:13.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:13 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:22:13.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:13 smithi016 ceph-mon[37215]: from='client.56447 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafe_smithi016-74264-33"}]': finished 2024-03-26T04:22:13.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:13 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-23", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:22:13.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:13 smithi016 ceph-mon[37215]: osdmap e208: 8 total, 8 up, 8 in 2024-03-26T04:22:13.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:13 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-23"}]: dispatch 2024-03-26T04:22:13.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:13 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3102021829' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi016-74285-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:13.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:13 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-23"}]: dispatch 2024-03-26T04:22:13.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:13 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1477746817' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi016-74264-34"}]: dispatch 2024-03-26T04:22:13.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:13 smithi016 ceph-mon[37215]: from='client.56465 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi016-74264-34"}]: dispatch 2024-03-26T04:22:13.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:13 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1477746817' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi016-74264-34"}]: dispatch 2024-03-26T04:22:13.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:13 smithi016 ceph-mon[37215]: from='client.56465 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi016-74264-34"}]: dispatch 2024-03-26T04:22:13.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:13 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1477746817' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi016-74264-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:13.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:13 smithi016 ceph-mon[37215]: from='client.56465 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi016-74264-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:13.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:13 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:13.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:13 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3102021829' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi016-74285-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:13.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:13 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-23"}]': finished 2024-03-26T04:22:13.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:13 smithi016 ceph-mon[37215]: from='client.56465 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValue_smithi016-74264-34", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:22:13.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:13 smithi016 ceph-mon[37215]: osdmap e209: 8 total, 8 up, 8 in 2024-03-26T04:22:13.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:13 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-23", "mode": "writeback"}]: dispatch 2024-03-26T04:22:13.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:13 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1477746817' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi016-74264-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi016-74264-34"}]: dispatch 2024-03-26T04:22:13.666 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:13 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-23", "mode": "writeback"}]: dispatch 2024-03-26T04:22:13.666 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:13 smithi016 ceph-mon[37215]: from='client.56465 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi016-74264-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi016-74264-34"}]: dispatch 2024-03-26T04:22:13.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:13 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1477746817' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi016-74264-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi016-74264-34"}]: dispatch 2024-03-26T04:22:13.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:13 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-23", "mode": "writeback"}]: dispatch 2024-03-26T04:22:13.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:13 smithi016 ceph-mon[29101]: from='client.56465 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValue_smithi016-74264-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi016-74264-34"}]: dispatch 2024-03-26T04:22:14.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:14 smithi183 ceph-mon[33745]: pgmap v272: 355 pgs: 32 unknown, 323 active+clean; 586 KiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T04:22:14.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:14 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:14.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:14 smithi183 ceph-mon[33745]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:22:14.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:14 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-23", "mode": "writeback"}]': finished 2024-03-26T04:22:14.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:14 smithi016 ceph-mon[29101]: pgmap v272: 355 pgs: 32 unknown, 323 active+clean; 586 KiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T04:22:14.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:14 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:14.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:14 smithi016 ceph-mon[29101]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:22:14.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:14 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-23", "mode": "writeback"}]': finished 2024-03-26T04:22:14.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:14 smithi016 ceph-mon[37215]: pgmap v272: 355 pgs: 32 unknown, 323 active+clean; 586 KiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T04:22:14.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:14 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:14.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:14 smithi016 ceph-mon[37215]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:22:14.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:14 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-23", "mode": "writeback"}]': finished 2024-03-26T04:22:15.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:15 smithi183 ceph-mon[33745]: osdmap e210: 8 total, 8 up, 8 in 2024-03-26T04:22:15.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:15 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2665593223' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi016-74285-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:15.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:15 smithi183 ceph-mon[33745]: from='client.56471 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi016-74285-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:15.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:15 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:15.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:15 smithi183 ceph-mon[33745]: from='client.56465 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi016-74264-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi016-74264-34"}]': finished 2024-03-26T04:22:15.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:15 smithi183 ceph-mon[33745]: from='client.56471 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi016-74285-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:15.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:15 smithi183 ceph-mon[33745]: osdmap e211: 8 total, 8 up, 8 in 2024-03-26T04:22:15.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:15 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:22:15.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:15 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:22:15.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:15 smithi016 ceph-mon[29101]: osdmap e210: 8 total, 8 up, 8 in 2024-03-26T04:22:15.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:15 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2665593223' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi016-74285-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:15.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:15 smithi016 ceph-mon[29101]: from='client.56471 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi016-74285-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:15.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:15 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:15.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:15 smithi016 ceph-mon[29101]: from='client.56465 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi016-74264-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi016-74264-34"}]': finished 2024-03-26T04:22:15.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:15 smithi016 ceph-mon[29101]: from='client.56471 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi016-74285-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:15.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:15 smithi016 ceph-mon[29101]: osdmap e211: 8 total, 8 up, 8 in 2024-03-26T04:22:15.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:15 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:22:15.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:15 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:22:15.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:15 smithi016 ceph-mon[37215]: osdmap e210: 8 total, 8 up, 8 in 2024-03-26T04:22:15.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:15 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2665593223' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi016-74285-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:15.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:15 smithi016 ceph-mon[37215]: from='client.56471 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi016-74285-40","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:15.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:15 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:15.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:15 smithi016 ceph-mon[37215]: from='client.56465 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValue_smithi016-74264-34", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValue_smithi016-74264-34"}]': finished 2024-03-26T04:22:15.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:15 smithi016 ceph-mon[37215]: from='client.56471 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi016-74285-40","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:15.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:15 smithi016 ceph-mon[37215]: osdmap e211: 8 total, 8 up, 8 in 2024-03-26T04:22:15.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:15 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:22:15.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:15 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:22:16.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:16 smithi183 ceph-mon[33745]: pgmap v275: 387 pgs: 64 unknown, 323 active+clean; 586 KiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T04:22:16.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:16 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:16.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:16 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:22:16.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:16 smithi183 ceph-mon[33745]: osdmap e212: 8 total, 8 up, 8 in 2024-03-26T04:22:16.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:16 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-23"}]: dispatch 2024-03-26T04:22:16.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:16 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2470664682' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi016-74285-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:16.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:16 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-23"}]: dispatch 2024-03-26T04:22:16.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:16 smithi183 ceph-mon[33745]: from='client.56477 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi016-74285-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:16.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:16 smithi016 ceph-mon[37215]: pgmap v275: 387 pgs: 64 unknown, 323 active+clean; 586 KiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T04:22:16.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:16 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:16.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:16 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:22:16.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:16 smithi016 ceph-mon[37215]: osdmap e212: 8 total, 8 up, 8 in 2024-03-26T04:22:16.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:16 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-23"}]: dispatch 2024-03-26T04:22:16.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:16 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2470664682' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi016-74285-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:16.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:16 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-23"}]: dispatch 2024-03-26T04:22:16.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:16 smithi016 ceph-mon[37215]: from='client.56477 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi016-74285-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:16.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:16 smithi016 ceph-mon[29101]: pgmap v275: 387 pgs: 64 unknown, 323 active+clean; 586 KiB data, 597 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T04:22:16.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:16 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:16.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:16 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:22:16.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:16 smithi016 ceph-mon[29101]: osdmap e212: 8 total, 8 up, 8 in 2024-03-26T04:22:16.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:16 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-23"}]: dispatch 2024-03-26T04:22:16.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:16 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2470664682' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi016-74285-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:16.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:16 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-23"}]: dispatch 2024-03-26T04:22:16.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:16 smithi016 ceph-mon[29101]: from='client.56477 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi016-74285-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:17.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:17 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:17.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:17 smithi183 ceph-mon[33745]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:22:17.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:17 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-23"}]': finished 2024-03-26T04:22:17.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:17 smithi183 ceph-mon[33745]: from='client.56477 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi016-74285-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:17.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:17 smithi183 ceph-mon[33745]: osdmap e213: 8 total, 8 up, 8 in 2024-03-26T04:22:17.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:17 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1477746817' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi016-74264-34"}]: dispatch 2024-03-26T04:22:17.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:17 smithi183 ceph-mon[33745]: from='client.56465 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi016-74264-34"}]: dispatch 2024-03-26T04:22:17.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:17 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:17.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:17 smithi016 ceph-mon[37215]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:22:17.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:17 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-23"}]': finished 2024-03-26T04:22:17.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:17 smithi016 ceph-mon[37215]: from='client.56477 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi016-74285-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:17.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:17 smithi016 ceph-mon[37215]: osdmap e213: 8 total, 8 up, 8 in 2024-03-26T04:22:17.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:17 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1477746817' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi016-74264-34"}]: dispatch 2024-03-26T04:22:17.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:17 smithi016 ceph-mon[37215]: from='client.56465 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi016-74264-34"}]: dispatch 2024-03-26T04:22:17.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:17 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:17.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:17 smithi016 ceph-mon[29101]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:22:17.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:17 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-23"}]': finished 2024-03-26T04:22:17.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:17 smithi016 ceph-mon[29101]: from='client.56477 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi016-74285-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:17.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:17 smithi016 ceph-mon[29101]: osdmap e213: 8 total, 8 up, 8 in 2024-03-26T04:22:17.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:17 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1477746817' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi016-74264-34"}]: dispatch 2024-03-26T04:22:17.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:17 smithi016 ceph-mon[29101]: from='client.56465 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi016-74264-34"}]: dispatch 2024-03-26T04:22:18.745 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:18 smithi016 ceph-mon[37215]: pgmap v278: 427 pgs: 3 creating+peering, 93 unknown, 331 active+clean; 586 KiB data, 597 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:22:18.745 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:18 smithi016 ceph-mon[37215]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:22:18.746 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:18 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:18.746 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:18 smithi016 ceph-mon[37215]: from='client.56465 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi016-74264-34"}]': finished 2024-03-26T04:22:18.746 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:18 smithi016 ceph-mon[37215]: osdmap e214: 8 total, 8 up, 8 in 2024-03-26T04:22:18.746 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:18 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1477746817' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi016-74264-34"}]: dispatch 2024-03-26T04:22:18.746 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:18 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/314638513' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi016-74285-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:18.746 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:18 smithi016 ceph-mon[37215]: from='client.56465 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi016-74264-34"}]: dispatch 2024-03-26T04:22:18.746 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:18 smithi016 ceph-mon[37215]: from='client.49012 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi016-74285-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:18.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:18 smithi016 ceph-mon[29101]: pgmap v278: 427 pgs: 3 creating+peering, 93 unknown, 331 active+clean; 586 KiB data, 597 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:22:18.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:18 smithi016 ceph-mon[29101]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:22:18.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:18 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:18.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:18 smithi016 ceph-mon[29101]: from='client.56465 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi016-74264-34"}]': finished 2024-03-26T04:22:18.747 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:18 smithi016 ceph-mon[29101]: osdmap e214: 8 total, 8 up, 8 in 2024-03-26T04:22:18.747 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:18 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1477746817' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi016-74264-34"}]: dispatch 2024-03-26T04:22:18.747 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:18 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/314638513' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi016-74285-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:18.747 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:18 smithi016 ceph-mon[29101]: from='client.56465 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi016-74264-34"}]: dispatch 2024-03-26T04:22:18.747 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:18 smithi016 ceph-mon[29101]: from='client.49012 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi016-74285-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:18.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:18 smithi183 ceph-mon[33745]: pgmap v278: 427 pgs: 3 creating+peering, 93 unknown, 331 active+clean; 586 KiB data, 597 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:22:18.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:18 smithi183 ceph-mon[33745]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:22:18.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:18 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:18.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:18 smithi183 ceph-mon[33745]: from='client.56465 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValue_smithi016-74264-34"}]': finished 2024-03-26T04:22:18.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:18 smithi183 ceph-mon[33745]: osdmap e214: 8 total, 8 up, 8 in 2024-03-26T04:22:18.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:18 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1477746817' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi016-74264-34"}]: dispatch 2024-03-26T04:22:18.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:18 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/314638513' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi016-74285-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:18.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:18 smithi183 ceph-mon[33745]: from='client.56465 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi016-74264-34"}]: dispatch 2024-03-26T04:22:18.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:18 smithi183 ceph-mon[33745]: from='client.49012 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi016-74285-42","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:19.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:22:18 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:22:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:22:19.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:19 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:19.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:19 smithi183 ceph-mon[33745]: from='client.56465 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi016-74264-34"}]': finished 2024-03-26T04:22:19.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:19 smithi183 ceph-mon[33745]: from='client.49012 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi016-74285-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:19.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:19 smithi183 ceph-mon[33745]: osdmap e215: 8 total, 8 up, 8 in 2024-03-26T04:22:19.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:19 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:19.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:19 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:19.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:19 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2867943621' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi016-74264-35"}]: dispatch 2024-03-26T04:22:19.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:19 smithi183 ceph-mon[33745]: from='client.49018 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi016-74264-35"}]: dispatch 2024-03-26T04:22:19.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:19 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2867943621' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi016-74264-35"}]: dispatch 2024-03-26T04:22:19.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:19 smithi183 ceph-mon[33745]: from='client.49018 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi016-74264-35"}]: dispatch 2024-03-26T04:22:19.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:19 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2867943621' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi016-74264-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:19.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:19 smithi183 ceph-mon[33745]: from='client.49018 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi016-74264-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:19.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:19 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:19.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:19 smithi016 ceph-mon[29101]: from='client.56465 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi016-74264-34"}]': finished 2024-03-26T04:22:19.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:19 smithi016 ceph-mon[29101]: from='client.49012 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi016-74285-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:19.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:19 smithi016 ceph-mon[29101]: osdmap e215: 8 total, 8 up, 8 in 2024-03-26T04:22:19.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:19 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:19.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:19 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:19.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:19 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2867943621' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi016-74264-35"}]: dispatch 2024-03-26T04:22:19.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:19 smithi016 ceph-mon[29101]: from='client.49018 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi016-74264-35"}]: dispatch 2024-03-26T04:22:19.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:19 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2867943621' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi016-74264-35"}]: dispatch 2024-03-26T04:22:19.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:19 smithi016 ceph-mon[29101]: from='client.49018 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi016-74264-35"}]: dispatch 2024-03-26T04:22:19.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:19 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2867943621' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi016-74264-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:19.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:19 smithi016 ceph-mon[29101]: from='client.49018 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi016-74264-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:19.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:19 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:19.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:19 smithi016 ceph-mon[37215]: from='client.56465 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValue_smithi016-74264-34"}]': finished 2024-03-26T04:22:19.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:19 smithi016 ceph-mon[37215]: from='client.49012 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi016-74285-42","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:19.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:19 smithi016 ceph-mon[37215]: osdmap e215: 8 total, 8 up, 8 in 2024-03-26T04:22:19.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:19 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:19.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:19 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-25","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:19.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:19 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2867943621' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi016-74264-35"}]: dispatch 2024-03-26T04:22:19.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:19 smithi016 ceph-mon[37215]: from='client.49018 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi016-74264-35"}]: dispatch 2024-03-26T04:22:19.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:19 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2867943621' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi016-74264-35"}]: dispatch 2024-03-26T04:22:19.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:19 smithi016 ceph-mon[37215]: from='client.49018 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi016-74264-35"}]: dispatch 2024-03-26T04:22:19.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:19 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2867943621' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi016-74264-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:19.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:19 smithi016 ceph-mon[37215]: from='client.49018 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi016-74264-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:20.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:20 smithi183 ceph-mon[33745]: pgmap v281: 419 pgs: 32 unknown, 32 creating+peering, 355 active+clean; 586 KiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:22:20.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:20 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:20.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:20 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:20.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:20 smithi183 ceph-mon[33745]: from='client.49018 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi016-74264-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:22:20.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:20 smithi183 ceph-mon[33745]: osdmap e216: 8 total, 8 up, 8 in 2024-03-26T04:22:20.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:20 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2867943621' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi016-74264-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi016-74264-35"}]: dispatch 2024-03-26T04:22:20.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:20 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1297871292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi016-74285-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:20.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:20 smithi183 ceph-mon[33745]: from='client.56495 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi016-74285-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:20.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:20 smithi183 ceph-mon[33745]: from='client.49018 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi016-74264-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi016-74264-35"}]: dispatch 2024-03-26T04:22:20.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:20 smithi016 ceph-mon[29101]: pgmap v281: 419 pgs: 32 unknown, 32 creating+peering, 355 active+clean; 586 KiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:22:20.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:20 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:20.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:20 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:20.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:20 smithi016 ceph-mon[29101]: from='client.49018 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi016-74264-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:22:20.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:20 smithi016 ceph-mon[29101]: osdmap e216: 8 total, 8 up, 8 in 2024-03-26T04:22:20.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:20 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2867943621' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi016-74264-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi016-74264-35"}]: dispatch 2024-03-26T04:22:20.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:20 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1297871292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi016-74285-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:20.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:20 smithi016 ceph-mon[29101]: from='client.56495 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi016-74285-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:20.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:20 smithi016 ceph-mon[29101]: from='client.49018 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi016-74264-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi016-74264-35"}]: dispatch 2024-03-26T04:22:20.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:20 smithi016 ceph-mon[37215]: pgmap v281: 419 pgs: 32 unknown, 32 creating+peering, 355 active+clean; 586 KiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:22:20.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:20 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:20.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:20 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-25","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:20.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:20 smithi016 ceph-mon[37215]: from='client.49018 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-Flush_smithi016-74264-35", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:22:20.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:20 smithi016 ceph-mon[37215]: osdmap e216: 8 total, 8 up, 8 in 2024-03-26T04:22:20.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:20 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2867943621' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi016-74264-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi016-74264-35"}]: dispatch 2024-03-26T04:22:20.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:20 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1297871292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi016-74285-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:20.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:20 smithi016 ceph-mon[37215]: from='client.56495 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi016-74285-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:20.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:20 smithi016 ceph-mon[37215]: from='client.49018 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "Flush_smithi016-74264-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi016-74264-35"}]: dispatch 2024-03-26T04:22:22.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:22 smithi016 ceph-mon[29101]: pgmap v284: 483 pgs: 96 unknown, 32 creating+peering, 355 active+clean; 586 KiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:22:22.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:22 smithi016 ceph-mon[29101]: from='client.56495 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi016-74285-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:22.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:22 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:22.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:22 smithi016 ceph-mon[29101]: osdmap e217: 8 total, 8 up, 8 in 2024-03-26T04:22:22.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:22 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:22:22.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:22 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:22:22.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:22 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:22:22.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:22 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:22:22.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:22 smithi016 ceph-mon[37215]: pgmap v284: 483 pgs: 96 unknown, 32 creating+peering, 355 active+clean; 586 KiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:22:22.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:22 smithi016 ceph-mon[37215]: from='client.56495 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi016-74285-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:22.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:22 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:22.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:22 smithi016 ceph-mon[37215]: osdmap e217: 8 total, 8 up, 8 in 2024-03-26T04:22:22.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:22 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:22:22.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:22 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:22:22.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:22 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:22:22.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:22 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:22:22.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:22:22 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:22:22.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:22 smithi183 ceph-mon[33745]: pgmap v284: 483 pgs: 96 unknown, 32 creating+peering, 355 active+clean; 586 KiB data, 581 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:22:22.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:22 smithi183 ceph-mon[33745]: from='client.56495 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RmXattrPP_smithi016-74285-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:22.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:22 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:22.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:22 smithi183 ceph-mon[33745]: osdmap e217: 8 total, 8 up, 8 in 2024-03-26T04:22:22.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:22 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:22:22.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:22 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-25", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:22:22.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:22 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:22:22.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:22 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:22:23.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:23 smithi016 ceph-mon[29101]: from='client.49018 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi016-74264-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi016-74264-35"}]': finished 2024-03-26T04:22:23.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:23 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-25", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:22:23.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:23 smithi016 ceph-mon[29101]: osdmap e218: 8 total, 8 up, 8 in 2024-03-26T04:22:23.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:23 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-25"}]: dispatch 2024-03-26T04:22:23.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:23 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-25"}]: dispatch 2024-03-26T04:22:23.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:23 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:23.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:23 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:22:23.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:23 smithi016 ceph-mon[37215]: from='client.49018 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi016-74264-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi016-74264-35"}]': finished 2024-03-26T04:22:23.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:23 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-25", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:22:23.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:23 smithi016 ceph-mon[37215]: osdmap e218: 8 total, 8 up, 8 in 2024-03-26T04:22:23.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:23 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-25"}]: dispatch 2024-03-26T04:22:23.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:23 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-25"}]: dispatch 2024-03-26T04:22:23.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:23 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:23.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:23 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:22:23.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:23 smithi183 ceph-mon[33745]: from='client.49018 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "Flush_smithi016-74264-35", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-Flush_smithi016-74264-35"}]': finished 2024-03-26T04:22:23.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:23 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-25", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:22:23.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:23 smithi183 ceph-mon[33745]: osdmap e218: 8 total, 8 up, 8 in 2024-03-26T04:22:23.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:23 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-25"}]: dispatch 2024-03-26T04:22:23.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:23 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-25"}]: dispatch 2024-03-26T04:22:23.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:23 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:23.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:23 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:22:24.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:24 smithi016 ceph-mon[37215]: pgmap v287: 459 pgs: 4 creating+peering, 4 unknown, 451 active+clean; 586 KiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:22:24.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:24 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-25"}]': finished 2024-03-26T04:22:24.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:24 smithi016 ceph-mon[37215]: osdmap e219: 8 total, 8 up, 8 in 2024-03-26T04:22:24.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:24 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-25", "mode": "writeback"}]: dispatch 2024-03-26T04:22:24.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:24 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-25", "mode": "writeback"}]: dispatch 2024-03-26T04:22:24.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:24 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:24.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:24 smithi016 ceph-mon[29101]: pgmap v287: 459 pgs: 4 creating+peering, 4 unknown, 451 active+clean; 586 KiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:22:24.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:24 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-25"}]': finished 2024-03-26T04:22:24.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:24 smithi016 ceph-mon[29101]: osdmap e219: 8 total, 8 up, 8 in 2024-03-26T04:22:24.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:24 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-25", "mode": "writeback"}]: dispatch 2024-03-26T04:22:24.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:24 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-25", "mode": "writeback"}]: dispatch 2024-03-26T04:22:24.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:24 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:24.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:24 smithi183 ceph-mon[33745]: pgmap v287: 459 pgs: 4 creating+peering, 4 unknown, 451 active+clean; 586 KiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:22:24.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:24 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-25"}]': finished 2024-03-26T04:22:24.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:24 smithi183 ceph-mon[33745]: osdmap e219: 8 total, 8 up, 8 in 2024-03-26T04:22:24.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:24 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-25", "mode": "writeback"}]: dispatch 2024-03-26T04:22:24.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:24 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-25", "mode": "writeback"}]: dispatch 2024-03-26T04:22:24.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:24 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:25.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:25 smithi016 ceph-mon[37215]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:22:25.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:25 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-25", "mode": "writeback"}]': finished 2024-03-26T04:22:25.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:25 smithi016 ceph-mon[37215]: osdmap e220: 8 total, 8 up, 8 in 2024-03-26T04:22:25.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:25 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2867943621' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi016-74264-35"}]: dispatch 2024-03-26T04:22:25.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:25 smithi016 ceph-mon[37215]: from='client.49018 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi016-74264-35"}]: dispatch 2024-03-26T04:22:25.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:25 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:25.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:25 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:22:25.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:25 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:22:25.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:25 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi016-74374-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-03-26T04:22:25.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:25 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi016-74374-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-03-26T04:22:25.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:25 smithi016 ceph-mon[29101]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:22:25.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:25 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-25", "mode": "writeback"}]': finished 2024-03-26T04:22:25.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:25 smithi016 ceph-mon[29101]: osdmap e220: 8 total, 8 up, 8 in 2024-03-26T04:22:25.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:25 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2867943621' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi016-74264-35"}]: dispatch 2024-03-26T04:22:25.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:25 smithi016 ceph-mon[29101]: from='client.49018 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi016-74264-35"}]: dispatch 2024-03-26T04:22:25.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:25 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:25.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:25 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:22:25.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:25 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:22:25.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:25 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi016-74374-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-03-26T04:22:25.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:25 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi016-74374-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-03-26T04:22:25.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:25 smithi183 ceph-mon[33745]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:22:25.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:25 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-25", "mode": "writeback"}]': finished 2024-03-26T04:22:25.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:25 smithi183 ceph-mon[33745]: osdmap e220: 8 total, 8 up, 8 in 2024-03-26T04:22:25.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:25 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2867943621' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi016-74264-35"}]: dispatch 2024-03-26T04:22:25.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:25 smithi183 ceph-mon[33745]: from='client.49018 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi016-74264-35"}]: dispatch 2024-03-26T04:22:25.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:25 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:25.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:25 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:22:25.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:25 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:22:25.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:25 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi016-74374-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-03-26T04:22:25.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:25 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi016-74374-1", "var": "pgp_num_actual", "val": "30"}]: dispatch 2024-03-26T04:22:26.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:26 smithi016 ceph-mon[29101]: pgmap v290: 387 pgs: 387 active+clean; 586 KiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:22:26.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:26 smithi016 ceph-mon[29101]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:22:26.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:26 smithi016 ceph-mon[29101]: from='client.49018 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi016-74264-35"}]': finished 2024-03-26T04:22:26.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:26 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:22:26.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:26 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi016-74374-1", "var": "pgp_num_actual", "val": "30"}]': finished 2024-03-26T04:22:26.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:26 smithi016 ceph-mon[29101]: osdmap e221: 8 total, 8 up, 8 in 2024-03-26T04:22:26.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:26 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-25"}]: dispatch 2024-03-26T04:22:26.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:26 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2867943621' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi016-74264-35"}]: dispatch 2024-03-26T04:22:26.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:26 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-25"}]: dispatch 2024-03-26T04:22:26.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:26 smithi016 ceph-mon[29101]: from='client.49018 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi016-74264-35"}]: dispatch 2024-03-26T04:22:26.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:26 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:26.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:26 smithi016 ceph-mon[37215]: pgmap v290: 387 pgs: 387 active+clean; 586 KiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:22:26.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:26 smithi016 ceph-mon[37215]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:22:26.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:26 smithi016 ceph-mon[37215]: from='client.49018 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi016-74264-35"}]': finished 2024-03-26T04:22:26.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:26 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:22:26.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:26 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi016-74374-1", "var": "pgp_num_actual", "val": "30"}]': finished 2024-03-26T04:22:26.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:26 smithi016 ceph-mon[37215]: osdmap e221: 8 total, 8 up, 8 in 2024-03-26T04:22:26.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:26 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-25"}]: dispatch 2024-03-26T04:22:26.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:26 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2867943621' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi016-74264-35"}]: dispatch 2024-03-26T04:22:26.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:26 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-25"}]: dispatch 2024-03-26T04:22:26.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:26 smithi016 ceph-mon[37215]: from='client.49018 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi016-74264-35"}]: dispatch 2024-03-26T04:22:26.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:26 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:26.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:26 smithi183 ceph-mon[33745]: pgmap v290: 387 pgs: 387 active+clean; 586 KiB data, 569 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:22:26.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:26 smithi183 ceph-mon[33745]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:22:26.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:26 smithi183 ceph-mon[33745]: from='client.49018 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-Flush_smithi016-74264-35"}]': finished 2024-03-26T04:22:26.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:26 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:22:26.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:26 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi016-74374-1", "var": "pgp_num_actual", "val": "30"}]': finished 2024-03-26T04:22:26.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:26 smithi183 ceph-mon[33745]: osdmap e221: 8 total, 8 up, 8 in 2024-03-26T04:22:26.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:26 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-25"}]: dispatch 2024-03-26T04:22:26.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:26 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2867943621' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi016-74264-35"}]: dispatch 2024-03-26T04:22:26.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:26 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-25"}]: dispatch 2024-03-26T04:22:26.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:26 smithi183 ceph-mon[33745]: from='client.49018 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"Flush_smithi016-74264-35"}]: dispatch 2024-03-26T04:22:26.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:26 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2332279674' entity='client.admin' cmd=[{"prefix":"status","format":"json"}]: dispatch 2024-03-26T04:22:27.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:27 smithi016 ceph-mon[37215]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:22:27.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:27 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-25"}]': finished 2024-03-26T04:22:27.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:27 smithi016 ceph-mon[37215]: from='client.49018 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi016-74264-35"}]': finished 2024-03-26T04:22:27.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:27 smithi016 ceph-mon[37215]: osdmap e222: 8 total, 8 up, 8 in 2024-03-26T04:22:27.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:27 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3016375588' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi016-74264-36"}]: dispatch 2024-03-26T04:22:27.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:27 smithi016 ceph-mon[37215]: from='client.56501 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi016-74264-36"}]: dispatch 2024-03-26T04:22:27.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:27 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3016375588' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi016-74264-36"}]: dispatch 2024-03-26T04:22:27.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:27 smithi016 ceph-mon[37215]: from='client.56501 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi016-74264-36"}]: dispatch 2024-03-26T04:22:27.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:27 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3016375588' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi016-74264-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:27.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:27 smithi016 ceph-mon[37215]: from='client.56501 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi016-74264-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:27.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:27 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi016-74374-1", "var": "pgp_num_actual", "val": "29"}]: dispatch 2024-03-26T04:22:27.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:27 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi016-74374-1", "var": "pgp_num_actual", "val": "29"}]: dispatch 2024-03-26T04:22:27.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:27 smithi016 ceph-mon[29101]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:22:27.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:27 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-25"}]': finished 2024-03-26T04:22:27.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:27 smithi016 ceph-mon[29101]: from='client.49018 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi016-74264-35"}]': finished 2024-03-26T04:22:27.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:27 smithi016 ceph-mon[29101]: osdmap e222: 8 total, 8 up, 8 in 2024-03-26T04:22:27.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:27 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3016375588' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi016-74264-36"}]: dispatch 2024-03-26T04:22:27.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:27 smithi016 ceph-mon[29101]: from='client.56501 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi016-74264-36"}]: dispatch 2024-03-26T04:22:27.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:27 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3016375588' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi016-74264-36"}]: dispatch 2024-03-26T04:22:27.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:27 smithi016 ceph-mon[29101]: from='client.56501 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi016-74264-36"}]: dispatch 2024-03-26T04:22:27.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:27 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3016375588' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi016-74264-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:27.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:27 smithi016 ceph-mon[29101]: from='client.56501 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi016-74264-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:27.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:27 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi016-74374-1", "var": "pgp_num_actual", "val": "29"}]: dispatch 2024-03-26T04:22:27.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:27 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi016-74374-1", "var": "pgp_num_actual", "val": "29"}]: dispatch 2024-03-26T04:22:27.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:27 smithi183 ceph-mon[33745]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:22:27.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:27 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-25"}]': finished 2024-03-26T04:22:27.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:27 smithi183 ceph-mon[33745]: from='client.49018 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"Flush_smithi016-74264-35"}]': finished 2024-03-26T04:22:27.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:27 smithi183 ceph-mon[33745]: osdmap e222: 8 total, 8 up, 8 in 2024-03-26T04:22:27.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:27 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3016375588' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi016-74264-36"}]: dispatch 2024-03-26T04:22:27.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:27 smithi183 ceph-mon[33745]: from='client.56501 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi016-74264-36"}]: dispatch 2024-03-26T04:22:27.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:27 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3016375588' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi016-74264-36"}]: dispatch 2024-03-26T04:22:27.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:27 smithi183 ceph-mon[33745]: from='client.56501 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi016-74264-36"}]: dispatch 2024-03-26T04:22:27.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:27 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3016375588' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi016-74264-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:27.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:27 smithi183 ceph-mon[33745]: from='client.56501 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi016-74264-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:27.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:27 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi016-74374-1", "var": "pgp_num_actual", "val": "29"}]: dispatch 2024-03-26T04:22:27.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:27 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "LibRadosList_smithi016-74374-1", "var": "pgp_num_actual", "val": "29"}]: dispatch 2024-03-26T04:22:28.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:28 smithi016 ceph-mon[29101]: pgmap v293: 323 pgs: 323 active+clean; 585 KiB data, 574 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:22:28.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:28 smithi016 ceph-mon[29101]: from='client.56501 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi016-74264-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:22:28.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:28 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi016-74374-1", "var": "pgp_num_actual", "val": "29"}]': finished 2024-03-26T04:22:28.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:28 smithi016 ceph-mon[29101]: osdmap e223: 8 total, 8 up, 8 in 2024-03-26T04:22:28.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:28 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3016375588' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi016-74264-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi016-74264-36"}]: dispatch 2024-03-26T04:22:28.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:28 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3833882523' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi016-74285-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:28.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:28 smithi016 ceph-mon[29101]: from='client.56501 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi016-74264-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi016-74264-36"}]: dispatch 2024-03-26T04:22:28.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:28 smithi016 ceph-mon[29101]: from='client.56507 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi016-74285-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:28.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:28 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/93714453' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi016-74374-2"}]: dispatch 2024-03-26T04:22:28.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:28 smithi016 ceph-mon[29101]: from='client.56513 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi016-74374-2"}]: dispatch 2024-03-26T04:22:28.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:28 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/93714453' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi016-74374-2"}]: dispatch 2024-03-26T04:22:28.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:28 smithi016 ceph-mon[29101]: from='client.56513 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi016-74374-2"}]: dispatch 2024-03-26T04:22:28.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:28 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/93714453' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi016-74374-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:28.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:28 smithi016 ceph-mon[29101]: from='client.56513 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi016-74374-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:28.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:28 smithi016 ceph-mon[37215]: pgmap v293: 323 pgs: 323 active+clean; 585 KiB data, 574 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:22:28.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:28 smithi016 ceph-mon[37215]: from='client.56501 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi016-74264-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:22:28.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:28 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi016-74374-1", "var": "pgp_num_actual", "val": "29"}]': finished 2024-03-26T04:22:28.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:28 smithi016 ceph-mon[37215]: osdmap e223: 8 total, 8 up, 8 in 2024-03-26T04:22:28.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:28 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3016375588' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi016-74264-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi016-74264-36"}]: dispatch 2024-03-26T04:22:28.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:28 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3833882523' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi016-74285-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:28.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:28 smithi016 ceph-mon[37215]: from='client.56501 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi016-74264-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi016-74264-36"}]: dispatch 2024-03-26T04:22:28.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:28 smithi016 ceph-mon[37215]: from='client.56507 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi016-74285-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:28.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:28 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/93714453' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi016-74374-2"}]: dispatch 2024-03-26T04:22:28.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:28 smithi016 ceph-mon[37215]: from='client.56513 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi016-74374-2"}]: dispatch 2024-03-26T04:22:28.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:28 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/93714453' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi016-74374-2"}]: dispatch 2024-03-26T04:22:28.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:28 smithi016 ceph-mon[37215]: from='client.56513 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi016-74374-2"}]: dispatch 2024-03-26T04:22:28.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:28 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/93714453' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi016-74374-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:28.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:28 smithi016 ceph-mon[37215]: from='client.56513 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi016-74374-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:28.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:28 smithi183 ceph-mon[33745]: pgmap v293: 323 pgs: 323 active+clean; 585 KiB data, 574 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:22:28.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:28 smithi183 ceph-mon[33745]: from='client.56501 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsync_smithi016-74264-36", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:22:28.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:28 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "LibRadosList_smithi016-74374-1", "var": "pgp_num_actual", "val": "29"}]': finished 2024-03-26T04:22:28.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:28 smithi183 ceph-mon[33745]: osdmap e223: 8 total, 8 up, 8 in 2024-03-26T04:22:28.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:28 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3016375588' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi016-74264-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi016-74264-36"}]: dispatch 2024-03-26T04:22:28.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:28 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3833882523' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi016-74285-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:28.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:28 smithi183 ceph-mon[33745]: from='client.56501 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsync_smithi016-74264-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi016-74264-36"}]: dispatch 2024-03-26T04:22:28.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:28 smithi183 ceph-mon[33745]: from='client.56507 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi016-74285-44","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:28.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:28 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/93714453' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi016-74374-2"}]: dispatch 2024-03-26T04:22:28.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:28 smithi183 ceph-mon[33745]: from='client.56513 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi016-74374-2"}]: dispatch 2024-03-26T04:22:28.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:28 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/93714453' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi016-74374-2"}]: dispatch 2024-03-26T04:22:28.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:28 smithi183 ceph-mon[33745]: from='client.56513 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi016-74374-2"}]: dispatch 2024-03-26T04:22:28.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:28 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/93714453' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi016-74374-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:28.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:28 smithi183 ceph-mon[33745]: from='client.56513 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi016-74374-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:29.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:22:28 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:22:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:22:29.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:29 smithi016 ceph-mon[37215]: from='client.56507 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi016-74285-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:29.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:29 smithi016 ceph-mon[37215]: from='client.56513 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi016-74374-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:22:29.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:29 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/93714453' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi016-74374-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi016-74374-2"}]: dispatch 2024-03-26T04:22:29.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:29 smithi016 ceph-mon[37215]: osdmap e224: 8 total, 8 up, 8 in 2024-03-26T04:22:29.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:29 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:29.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:29 smithi016 ceph-mon[37215]: from='client.56513 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi016-74374-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi016-74374-2"}]: dispatch 2024-03-26T04:22:29.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:29 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:29.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:29 smithi016 ceph-mon[37215]: from='client.56501 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi016-74264-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi016-74264-36"}]': finished 2024-03-26T04:22:29.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:29 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:29.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:29 smithi016 ceph-mon[37215]: osdmap e225: 8 total, 8 up, 8 in 2024-03-26T04:22:29.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:29 smithi016 ceph-mon[29101]: from='client.56507 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi016-74285-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:29.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:29 smithi016 ceph-mon[29101]: from='client.56513 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi016-74374-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:22:29.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:29 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/93714453' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi016-74374-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi016-74374-2"}]: dispatch 2024-03-26T04:22:29.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:29 smithi016 ceph-mon[29101]: osdmap e224: 8 total, 8 up, 8 in 2024-03-26T04:22:29.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:29 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:29.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:29 smithi016 ceph-mon[29101]: from='client.56513 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi016-74374-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi016-74374-2"}]: dispatch 2024-03-26T04:22:29.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:29 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:29.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:29 smithi016 ceph-mon[29101]: from='client.56501 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi016-74264-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi016-74264-36"}]': finished 2024-03-26T04:22:29.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:29 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:29.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:29 smithi016 ceph-mon[29101]: osdmap e225: 8 total, 8 up, 8 in 2024-03-26T04:22:29.710 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:29 smithi183 ceph-mon[33745]: from='client.56507 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RemoveTestPP_smithi016-74285-44","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:29.710 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:29 smithi183 ceph-mon[33745]: from='client.56513 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosListEC_smithi016-74374-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:22:29.710 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:29 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/93714453' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi016-74374-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi016-74374-2"}]: dispatch 2024-03-26T04:22:29.710 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:29 smithi183 ceph-mon[33745]: osdmap e224: 8 total, 8 up, 8 in 2024-03-26T04:22:29.711 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:29 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:29.711 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:29 smithi183 ceph-mon[33745]: from='client.56513 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi016-74374-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi016-74374-2"}]: dispatch 2024-03-26T04:22:29.711 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:29 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-27","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:29.711 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:29 smithi183 ceph-mon[33745]: from='client.56501 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsync_smithi016-74264-36", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsync_smithi016-74264-36"}]': finished 2024-03-26T04:22:29.711 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:29 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-27","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:29.711 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:29 smithi183 ceph-mon[33745]: osdmap e225: 8 total, 8 up, 8 in 2024-03-26T04:22:30.682 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:30 smithi183 ceph-mon[33745]: pgmap v296: 324 pgs: 7 creating+peering, 57 unknown, 260 active+clean; 583 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:22:30.682 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:30 smithi183 ceph-mon[33745]: from='client.56513 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi016-74374-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi016-74374-2"}]': finished 2024-03-26T04:22:30.682 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:30 smithi183 ceph-mon[33745]: osdmap e226: 8 total, 8 up, 8 in 2024-03-26T04:22:30.682 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:30 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1799010161' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi016-74285-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:30.682 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:30 smithi183 ceph-mon[33745]: from='client.56519 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi016-74285-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:30.682 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:30 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:22:30.683 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:30 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:22:30.683 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:30 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:22:30.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:30 smithi016 ceph-mon[37215]: pgmap v296: 324 pgs: 7 creating+peering, 57 unknown, 260 active+clean; 583 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:22:30.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:30 smithi016 ceph-mon[37215]: from='client.56513 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi016-74374-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi016-74374-2"}]': finished 2024-03-26T04:22:30.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:30 smithi016 ceph-mon[37215]: osdmap e226: 8 total, 8 up, 8 in 2024-03-26T04:22:30.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:30 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1799010161' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi016-74285-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:30.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:30 smithi016 ceph-mon[37215]: from='client.56519 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi016-74285-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:30.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:30 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:22:30.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:30 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:22:30.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:30 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:22:30.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:30 smithi016 ceph-mon[29101]: pgmap v296: 324 pgs: 7 creating+peering, 57 unknown, 260 active+clean; 583 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:22:30.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:30 smithi016 ceph-mon[29101]: from='client.56513 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosListEC_smithi016-74374-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosListEC_smithi016-74374-2"}]': finished 2024-03-26T04:22:30.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:30 smithi016 ceph-mon[29101]: osdmap e226: 8 total, 8 up, 8 in 2024-03-26T04:22:30.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:30 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1799010161' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi016-74285-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:30.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:30 smithi016 ceph-mon[29101]: from='client.56519 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi016-74285-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:30.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:30 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:22:30.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:30 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-27", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:22:30.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:30 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:22:31.210 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAintry=2 expected=2 2024-03-26T04:22:31.210 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : seek to 16:e960b815:::9:head 2024-03-26T04:22:31.211 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : cursor()=16:e960b815:::9:head expected=16:e960b815:::9:head 2024-03-26T04:22:31.211 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > 16:e960b815:::9:head -> 9 2024-03-26T04:22:31.211 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : entry=9 expected=9 2024-03-26T04:22:31.211 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : seek to 16:de5d7c5f:::12:head 2024-03-26T04:22:31.211 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : cursor()=16:de5d7c5f:::12:head expected=16:de5d7c5f:::12:head 2024-03-26T04:22:31.211 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > 16:de5d7c5f:::12:head -> 12 2024-03-26T04:22:31.212 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : entry=12 expected=12 2024-03-26T04:22:31.212 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : seek to 16:02547ec2:::1:head 2024-03-26T04:22:31.212 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : cursor()=16:02547ec2:::1:head expected=16:02547ec2:::1:head 2024-03-26T04:22:31.212 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > 16:02547ec2:::1:head -> 1 2024-03-26T04:22:31.212 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : entry=1 expected=1 2024-03-26T04:22:31.212 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : seek to 16:bd63b0f1:::8:head 2024-03-26T04:22:31.212 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : cursor()=16:bd63b0f1:::8:head expected=16:bd63b0f1:::8:head 2024-03-26T04:22:31.213 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > 16:bd63b0f1:::8:head -> 8 2024-03-26T04:22:31.213 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : entry=8 expected=8 2024-03-26T04:22:31.213 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : seek to 16:b29083e3:::5:head 2024-03-26T04:22:31.213 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : cursor()=16:b29083e3:::5:head expected=16:b29083e3:::5:head 2024-03-26T04:22:31.213 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > 16:b29083e3:::5:head -> 5 2024-03-26T04:22:31.213 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : entry=5 expected=5 2024-03-26T04:22:31.213 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : seek to 16:89d3ae78:::11:head 2024-03-26T04:22:31.214 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : cursor()=16:89d3ae78:::11:head expected=16:89d3ae78:::11:head 2024-03-26T04:22:31.214 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > 16:89d3ae78:::11:head -> 11 2024-03-26T04:22:31.214 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : entry=11 expected=11 2024-03-26T04:22:31.214 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : seek to 16:52ea6a34:::10:head 2024-03-26T04:22:31.214 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : cursor()=16:52ea6a34:::10:head expected=16:52ea6a34:::10:head 2024-03-26T04:22:31.214 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > 16:52ea6a34:::10:head -> 10 2024-03-26T04:22:31.215 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : entry=10 expected=10 2024-03-26T04:22:31.215 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : seek to 16:5c6b0b28:::7:head 2024-03-26T04:22:31.215 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : cursor()=16:5c6b0b28:::7:head expected=16:5c6b0b28:::7:head 2024-03-26T04:22:31.215 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: > 16:5c6b0b28:::7:head -> 7 2024-03-26T04:22:31.215 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: : entry=7 expected=7 2024-03-26T04:22:31.215 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (258 ms) 2024-03-26T04:22:31.215 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-03-26T04:22:31.216 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (115090 ms) 2024-03-26T04:22:31.216 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-03-26T04:22:31.216 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: split 0/5 -> MIN 16:33333333::::head 2024-03-26T04:22:31.216 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: split 1/5 -> 16:33333333::::head 16:66666666::::head 2024-03-26T04:22:31.216 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: split 2/5 -> 16:66666666::::head 16:99999999::::head 2024-03-26T04:22:31.216 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: split 3/5 -> 16:99999999::::head 16:cccccccc::::head 2024-03-26T04:22:31.216 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: split 4/5 -> 16:cccccccc::::head MAX 2024-03-26T04:22:31.216 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (32023 ms) 2024-03-26T04:22:31.217 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: [----------] 7 tests from LibRadosList (147958 ms total) 2024-03-26T04:22:31.217 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: 2024-03-26T04:22:31.217 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-03-26T04:22:31.217 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-03-26T04:22:31.217 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1032 ms) 2024-03-26T04:22:31.217 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-03-26T04:22:31.217 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: myset foo1,foo2,foo3 2024-03-26T04:22:31.217 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: foo1 2024-03-26T04:22:31.217 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: foo2 2024-03-26T04:22:31.218 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: foo3 2024-03-26T04:22:31.218 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: myset foo1,foo4,foo5 2024-03-26T04:22:31.218 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: foo4 2024-03-26T04:22:31.218 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: foo5 2024-03-26T04:22:31.218 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: foo1 2024-03-26T04:22:31.218 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: myset foo6,foo7 2024-03-26T04:22:31.218 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: foo7 2024-03-26T04:22:31.218 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: foo6 2024-03-26T04:22:31.218 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-03-26T04:22:31.218 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: ns1:foo4 2024-03-26T04:22:31.218 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: ns1:foo5 2024-03-26T04:22:31.218 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: ns2:foo7 2024-03-26T04:22:31.219 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: ns2:foo6 2024-03-26T04:22:31.219 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: ns1:foo1 2024-03-26T04:22:31.219 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: :foo1 2024-03-26T04:22:31.219 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: :foo2 2024-03-26T04:22:31.219 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: :foo3 2024-03-26T04:22:31.219 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (30 ms) 2024-03-26T04:22:31.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:31 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:22:31.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:31 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:22:31.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:31 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:22:31.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:31 smithi183 ceph-mon[33745]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:22:31.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:31 smithi183 ceph-mon[33745]: from='client.56519 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi016-74285-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:31.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:31 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-27", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:22:31.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:31 smithi183 ceph-mon[33745]: osdmap e227: 8 total, 8 up, 8 in 2024-03-26T04:22:31.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:31 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-27"}]: dispatch 2024-03-26T04:22:31.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:31 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3016375588' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi016-74264-36"}]: dispatch 2024-03-26T04:22:31.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:31 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-27"}]: dispatch 2024-03-26T04:22:31.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:31 smithi183 ceph-mon[33745]: from='client.56501 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi016-74264-36"}]: dispatch 2024-03-26T04:22:31.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:31 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:22:31.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:31 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:22:31.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:31 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:22:31.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:31 smithi016 ceph-mon[37215]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:22:31.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:31 smithi016 ceph-mon[37215]: from='client.56519 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi016-74285-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:31.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:31 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-27", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:22:31.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:31 smithi016 ceph-mon[37215]: osdmap e227: 8 total, 8 up, 8 in 2024-03-26T04:22:31.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:31 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-27"}]: dispatch 2024-03-26T04:22:31.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:31 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3016375588' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi016-74264-36"}]: dispatch 2024-03-26T04:22:31.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:31 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-27"}]: dispatch 2024-03-26T04:22:31.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:31 smithi016 ceph-mon[37215]: from='client.56501 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi016-74264-36"}]: dispatch 2024-03-26T04:22:31.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:31 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:22:31.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:31 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:22:31.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:31 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:22:31.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:31 smithi016 ceph-mon[29101]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:22:31.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:31 smithi016 ceph-mon[29101]: from='client.56519 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "XattrListPP_smithi016-74285-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:31.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:31 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-27", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:22:31.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:31 smithi016 ceph-mon[29101]: osdmap e227: 8 total, 8 up, 8 in 2024-03-26T04:22:31.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:31 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-27"}]: dispatch 2024-03-26T04:22:31.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:31 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3016375588' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi016-74264-36"}]: dispatch 2024-03-26T04:22:31.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:31 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-27"}]: dispatch 2024-03-26T04:22:31.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:31 smithi016 ceph-mon[29101]: from='client.56501 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi016-74264-36"}]: dispatch 2024-03-26T04:22:32.542 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:22:32 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:22:32.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:32 smithi183 ceph-mon[33745]: pgmap v299: 340 pgs: 80 unknown, 260 active+clean; 583 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:22:32.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:32 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-27"}]': finished 2024-03-26T04:22:32.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:32 smithi183 ceph-mon[33745]: from='client.56501 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi016-74264-36"}]': finished 2024-03-26T04:22:32.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:32 smithi183 ceph-mon[33745]: osdmap e228: 8 total, 8 up, 8 in 2024-03-26T04:22:32.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:32 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-27", "mode": "writeback"}]: dispatch 2024-03-26T04:22:32.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:32 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3016375588' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi016-74264-36"}]: dispatch 2024-03-26T04:22:32.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:32 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-27", "mode": "writeback"}]: dispatch 2024-03-26T04:22:32.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:32 smithi183 ceph-mon[33745]: from='client.56501 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi016-74264-36"}]: dispatch 2024-03-26T04:22:32.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:32 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/93714453' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi016-74374-2"}]: dispatch 2024-03-26T04:22:32.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:32 smithi183 ceph-mon[33745]: from='client.56513 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi016-74374-2"}]: dispatch 2024-03-26T04:22:32.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:32 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/4195605130' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi016-74285-46"}]: dispatch 2024-03-26T04:22:32.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:32 smithi183 ceph-mon[33745]: from='client.49048 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi016-74285-46"}]: dispatch 2024-03-26T04:22:32.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:32 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/4195605130' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi016-74285-46"}]: dispatch 2024-03-26T04:22:32.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:32 smithi183 ceph-mon[33745]: from='client.49048 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi016-74285-46"}]: dispatch 2024-03-26T04:22:32.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:32 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/4195605130' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi016-74285-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:32.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:32 smithi183 ceph-mon[33745]: from='client.49048 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi016-74285-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:32.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:32 smithi016 ceph-mon[29101]: pgmap v299: 340 pgs: 80 unknown, 260 active+clean; 583 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:22:32.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:32 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-27"}]': finished 2024-03-26T04:22:32.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:32 smithi016 ceph-mon[29101]: from='client.56501 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi016-74264-36"}]': finished 2024-03-26T04:22:32.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:32 smithi016 ceph-mon[29101]: osdmap e228: 8 total, 8 up, 8 in 2024-03-26T04:22:32.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:32 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-27", "mode": "writeback"}]: dispatch 2024-03-26T04:22:32.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:32 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3016375588' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi016-74264-36"}]: dispatch 2024-03-26T04:22:32.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:32 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-27", "mode": "writeback"}]: dispatch 2024-03-26T04:22:32.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:32 smithi016 ceph-mon[29101]: from='client.56501 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi016-74264-36"}]: dispatch 2024-03-26T04:22:32.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:32 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/93714453' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi016-74374-2"}]: dispatch 2024-03-26T04:22:32.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:32 smithi016 ceph-mon[29101]: from='client.56513 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi016-74374-2"}]: dispatch 2024-03-26T04:22:32.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:32 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/4195605130' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi016-74285-46"}]: dispatch 2024-03-26T04:22:32.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:32 smithi016 ceph-mon[29101]: from='client.49048 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi016-74285-46"}]: dispatch 2024-03-26T04:22:32.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:32 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/4195605130' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi016-74285-46"}]: dispatch 2024-03-26T04:22:32.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:32 smithi016 ceph-mon[29101]: from='client.49048 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi016-74285-46"}]: dispatch 2024-03-26T04:22:32.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:32 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/4195605130' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi016-74285-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:32.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:32 smithi016 ceph-mon[29101]: from='client.49048 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi016-74285-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:32.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:32 smithi016 ceph-mon[37215]: pgmap v299: 340 pgs: 80 unknown, 260 active+clean; 583 KiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:22:32.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:32 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-27"}]': finished 2024-03-26T04:22:32.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:32 smithi016 ceph-mon[37215]: from='client.56501 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsync_smithi016-74264-36"}]': finished 2024-03-26T04:22:32.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:32 smithi016 ceph-mon[37215]: osdmap e228: 8 total, 8 up, 8 in 2024-03-26T04:22:32.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:32 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-27", "mode": "writeback"}]: dispatch 2024-03-26T04:22:32.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:32 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3016375588' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi016-74264-36"}]: dispatch 2024-03-26T04:22:32.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:32 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-27", "mode": "writeback"}]: dispatch 2024-03-26T04:22:32.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:32 smithi016 ceph-mon[37215]: from='client.56501 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi016-74264-36"}]: dispatch 2024-03-26T04:22:32.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:32 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/93714453' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi016-74374-2"}]: dispatch 2024-03-26T04:22:32.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:32 smithi016 ceph-mon[37215]: from='client.56513 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi016-74374-2"}]: dispatch 2024-03-26T04:22:32.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:32 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/4195605130' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi016-74285-46"}]: dispatch 2024-03-26T04:22:32.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:32 smithi016 ceph-mon[37215]: from='client.49048 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi016-74285-46"}]: dispatch 2024-03-26T04:22:32.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:32 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/4195605130' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi016-74285-46"}]: dispatch 2024-03-26T04:22:32.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:32 smithi016 ceph-mon[37215]: from='client.49048 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi016-74285-46"}]: dispatch 2024-03-26T04:22:32.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:32 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/4195605130' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi016-74285-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:32.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:32 smithi016 ceph-mon[37215]: from='client.49048 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi016-74285-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:33.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:33 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:22:33.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:33 smithi183 ceph-mon[33745]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:22:33.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:33 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-27", "mode": "writeback"}]': finished 2024-03-26T04:22:33.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:33 smithi183 ceph-mon[33745]: from='client.56501 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi016-74264-36"}]': finished 2024-03-26T04:22:33.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:33 smithi183 ceph-mon[33745]: from='client.56513 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi016-74374-2"}]': finished 2024-03-26T04:22:33.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:33 smithi183 ceph-mon[33745]: from='client.49048 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi016-74285-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:22:33.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:33 smithi183 ceph-mon[33745]: osdmap e229: 8 total, 8 up, 8 in 2024-03-26T04:22:33.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:33 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/93714453' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi016-74374-2"}]: dispatch 2024-03-26T04:22:33.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:33 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/4195605130' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi016-74285-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi016-74285-46"}]: dispatch 2024-03-26T04:22:33.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:33 smithi183 ceph-mon[33745]: from='client.56513 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi016-74374-2"}]: dispatch 2024-03-26T04:22:33.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:33 smithi183 ceph-mon[33745]: from='client.49048 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi016-74285-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi016-74285-46"}]: dispatch 2024-03-26T04:22:33.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:33 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/266303643' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi016-74264-37"}]: dispatch 2024-03-26T04:22:33.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:33 smithi183 ceph-mon[33745]: from='client.49054 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi016-74264-37"}]: dispatch 2024-03-26T04:22:33.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:33 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/266303643' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi016-74264-37"}]: dispatch 2024-03-26T04:22:33.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:33 smithi183 ceph-mon[33745]: from='client.49054 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi016-74264-37"}]: dispatch 2024-03-26T04:22:33.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:33 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/266303643' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi016-74264-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:33.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:33 smithi183 ceph-mon[33745]: from='client.49054 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi016-74264-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:33.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:33 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:22:33.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:33 smithi016 ceph-mon[37215]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:22:33.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:33 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-27", "mode": "writeback"}]': finished 2024-03-26T04:22:33.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:33 smithi016 ceph-mon[37215]: from='client.56501 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi016-74264-36"}]': finished 2024-03-26T04:22:33.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:33 smithi016 ceph-mon[37215]: from='client.56513 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi016-74374-2"}]': finished 2024-03-26T04:22:33.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:33 smithi016 ceph-mon[37215]: from='client.49048 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi016-74285-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:22:33.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:33 smithi016 ceph-mon[37215]: osdmap e229: 8 total, 8 up, 8 in 2024-03-26T04:22:33.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:33 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/93714453' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi016-74374-2"}]: dispatch 2024-03-26T04:22:33.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:33 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/4195605130' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi016-74285-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi016-74285-46"}]: dispatch 2024-03-26T04:22:33.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:33 smithi016 ceph-mon[37215]: from='client.56513 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi016-74374-2"}]: dispatch 2024-03-26T04:22:33.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:33 smithi016 ceph-mon[37215]: from='client.49048 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi016-74285-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi016-74285-46"}]: dispatch 2024-03-26T04:22:33.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:33 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/266303643' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi016-74264-37"}]: dispatch 2024-03-26T04:22:33.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:33 smithi016 ceph-mon[37215]: from='client.49054 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi016-74264-37"}]: dispatch 2024-03-26T04:22:33.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:33 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/266303643' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi016-74264-37"}]: dispatch 2024-03-26T04:22:33.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:33 smithi016 ceph-mon[37215]: from='client.49054 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi016-74264-37"}]: dispatch 2024-03-26T04:22:33.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:33 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/266303643' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi016-74264-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:33.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:33 smithi016 ceph-mon[37215]: from='client.49054 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi016-74264-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:33.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:33 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:22:33.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:33 smithi016 ceph-mon[29101]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:22:33.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:33 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-27", "mode": "writeback"}]': finished 2024-03-26T04:22:33.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:33 smithi016 ceph-mon[29101]: from='client.56501 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsync_smithi016-74264-36"}]': finished 2024-03-26T04:22:33.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:33 smithi016 ceph-mon[29101]: from='client.56513 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosListEC_smithi016-74374-2"}]': finished 2024-03-26T04:22:33.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:33 smithi016 ceph-mon[29101]: from='client.49048 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi016-74285-46", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:22:33.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:33 smithi016 ceph-mon[29101]: osdmap e229: 8 total, 8 up, 8 in 2024-03-26T04:22:33.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:33 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/93714453' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi016-74374-2"}]: dispatch 2024-03-26T04:22:33.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:33 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/4195605130' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi016-74285-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi016-74285-46"}]: dispatch 2024-03-26T04:22:33.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:33 smithi016 ceph-mon[29101]: from='client.56513 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi016-74374-2"}]: dispatch 2024-03-26T04:22:33.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:33 smithi016 ceph-mon[29101]: from='client.49048 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi016-74285-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi016-74285-46"}]: dispatch 2024-03-26T04:22:33.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:33 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/266303643' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi016-74264-37"}]: dispatch 2024-03-26T04:22:33.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:33 smithi016 ceph-mon[29101]: from='client.49054 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi016-74264-37"}]: dispatch 2024-03-26T04:22:33.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:33 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/266303643' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi016-74264-37"}]: dispatch 2024-03-26T04:22:33.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:33 smithi016 ceph-mon[29101]: from='client.49054 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi016-74264-37"}]: dispatch 2024-03-26T04:22:33.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:33 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/266303643' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi016-74264-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:33.916 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:33 smithi016 ceph-mon[29101]: from='client.49054 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi016-74264-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:34.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:34 smithi016 ceph-mon[29101]: pgmap v302: 292 pgs: 292 active+clean; 584 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T04:22:34.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:34 smithi016 ceph-mon[29101]: from='client.56513 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi016-74374-2"}]': finished 2024-03-26T04:22:34.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:34 smithi016 ceph-mon[29101]: from='client.49054 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi016-74264-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:22:34.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:34 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/266303643' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi016-74264-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi016-74264-37"}]: dispatch 2024-03-26T04:22:34.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:34 smithi016 ceph-mon[29101]: osdmap e230: 8 total, 8 up, 8 in 2024-03-26T04:22:34.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:34 smithi016 ceph-mon[29101]: from='client.49054 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi016-74264-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi016-74264-37"}]: dispatch 2024-03-26T04:22:34.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:34 smithi016 ceph-mon[37215]: pgmap v302: 292 pgs: 292 active+clean; 584 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T04:22:34.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:34 smithi016 ceph-mon[37215]: from='client.56513 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi016-74374-2"}]': finished 2024-03-26T04:22:34.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:34 smithi016 ceph-mon[37215]: from='client.49054 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi016-74264-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:22:34.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:34 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/266303643' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi016-74264-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi016-74264-37"}]: dispatch 2024-03-26T04:22:34.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:34 smithi016 ceph-mon[37215]: osdmap e230: 8 total, 8 up, 8 in 2024-03-26T04:22:34.915 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:34 smithi016 ceph-mon[37215]: from='client.49054 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi016-74264-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi016-74264-37"}]: dispatch 2024-03-26T04:22:35.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:34 smithi183 ceph-mon[33745]: pgmap v302: 292 pgs: 292 active+clean; 584 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T04:22:35.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:34 smithi183 ceph-mon[33745]: from='client.56513 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosListEC_smithi016-74374-2"}]': finished 2024-03-26T04:22:35.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:34 smithi183 ceph-mon[33745]: from='client.49054 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFull_smithi016-74264-37", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:22:35.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:34 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/266303643' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi016-74264-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi016-74264-37"}]: dispatch 2024-03-26T04:22:35.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:34 smithi183 ceph-mon[33745]: osdmap e230: 8 total, 8 up, 8 in 2024-03-26T04:22:35.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:34 smithi183 ceph-mon[33745]: from='client.49054 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi016-74264-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi016-74264-37"}]: dispatch 2024-03-26T04:22:35.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:35 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:22:35.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:35 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:22:35.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:35 smithi016 ceph-mon[37215]: pgmap v305: 292 pgs: 292 active+clean; 584 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T04:22:35.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:35 smithi016 ceph-mon[37215]: from='client.49048 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi016-74285-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi016-74285-46"}]': finished 2024-03-26T04:22:35.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:35 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:22:35.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:35 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-27"}]: dispatch 2024-03-26T04:22:35.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:35 smithi016 ceph-mon[37215]: osdmap e231: 8 total, 8 up, 8 in 2024-03-26T04:22:35.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:35 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2547925361' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74374-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:35.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:35 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-27"}]: dispatch 2024-03-26T04:22:35.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:35 smithi016 ceph-mon[37215]: from='client.56537 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74374-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:35.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:35 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:22:35.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:35 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:22:35.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:35 smithi016 ceph-mon[29101]: pgmap v305: 292 pgs: 292 active+clean; 584 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T04:22:35.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:35 smithi016 ceph-mon[29101]: from='client.49048 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi016-74285-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi016-74285-46"}]': finished 2024-03-26T04:22:35.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:35 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:22:35.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:35 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-27"}]: dispatch 2024-03-26T04:22:35.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:35 smithi016 ceph-mon[29101]: osdmap e231: 8 total, 8 up, 8 in 2024-03-26T04:22:35.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:35 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2547925361' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74374-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:35.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:35 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-27"}]: dispatch 2024-03-26T04:22:35.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:35 smithi016 ceph-mon[29101]: from='client.56537 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74374-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:36.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:35 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:22:36.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:35 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:22:36.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:35 smithi183 ceph-mon[33745]: pgmap v305: 292 pgs: 292 active+clean; 584 KiB data, 563 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T04:22:36.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:35 smithi183 ceph-mon[33745]: from='client.49048 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi016-74285-46", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi016-74285-46"}]': finished 2024-03-26T04:22:36.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:35 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:22:36.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:35 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-27"}]: dispatch 2024-03-26T04:22:36.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:35 smithi183 ceph-mon[33745]: osdmap e231: 8 total, 8 up, 8 in 2024-03-26T04:22:36.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:35 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2547925361' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74374-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:36.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:35 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-27"}]: dispatch 2024-03-26T04:22:36.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:35 smithi183 ceph-mon[33745]: from='client.56537 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74374-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:37.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:36 smithi183 ceph-mon[33745]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:22:37.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:36 smithi183 ceph-mon[33745]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:22:37.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:36 smithi183 ceph-mon[33745]: from='client.49054 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi016-74264-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi016-74264-37"}]': finished 2024-03-26T04:22:37.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:36 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-27"}]': finished 2024-03-26T04:22:37.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:36 smithi183 ceph-mon[33745]: from='client.56537 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74374-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:37.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:36 smithi183 ceph-mon[33745]: osdmap e232: 8 total, 8 up, 8 in 2024-03-26T04:22:37.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:36 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2547925361' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi016-74374-3","pool2":"test-rados-api-smithi016-74374-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-26T04:22:37.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:36 smithi183 ceph-mon[33745]: from='client.56537 ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi016-74374-3","pool2":"test-rados-api-smithi016-74374-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-26T04:22:37.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:36 smithi016 ceph-mon[29101]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:22:37.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:36 smithi016 ceph-mon[29101]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:22:37.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:36 smithi016 ceph-mon[29101]: from='client.49054 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi016-74264-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi016-74264-37"}]': finished 2024-03-26T04:22:37.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:36 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-27"}]': finished 2024-03-26T04:22:37.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:36 smithi016 ceph-mon[29101]: from='client.56537 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74374-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:37.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:36 smithi016 ceph-mon[29101]: osdmap e232: 8 total, 8 up, 8 in 2024-03-26T04:22:37.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:36 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2547925361' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi016-74374-3","pool2":"test-rados-api-smithi016-74374-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-26T04:22:37.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:36 smithi016 ceph-mon[29101]: from='client.56537 ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi016-74374-3","pool2":"test-rados-api-smithi016-74374-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-26T04:22:37.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:36 smithi016 ceph-mon[37215]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:22:37.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:36 smithi016 ceph-mon[37215]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:22:37.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:36 smithi016 ceph-mon[37215]: from='client.49054 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFull_smithi016-74264-37", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFull_smithi016-74264-37"}]': finished 2024-03-26T04:22:37.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:36 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-27"}]': finished 2024-03-26T04:22:37.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:36 smithi016 ceph-mon[37215]: from='client.56537 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74374-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:37.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:36 smithi016 ceph-mon[37215]: osdmap e232: 8 total, 8 up, 8 in 2024-03-26T04:22:37.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:36 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2547925361' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi016-74374-3","pool2":"test-rados-api-smithi016-74374-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-26T04:22:37.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:36 smithi016 ceph-mon[37215]: from='client.56537 ' entity='client.admin' cmd=[{"prefix":"osd pool rm","pool": "test-rados-api-smithi016-74374-3","pool2":"test-rados-api-smithi016-74374-3","yes_i_really_really_mean_it_not_faking": true}]: dispatch 2024-03-26T04:22:37.164 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-03-26T04:22:37.164 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: 1 0 2024-03-26T04:22:37.165 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: 10 0 2024-03-26T04:22:37.165 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: 13 0 2024-03-26T04:22:37.165 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: 7 0 2024-03-26T04:22:37.165 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: 14 0 2024-03-26T04:22:37.165 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: 0 0 2024-03-26T04:22:37.165 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: 15 0 2024-03-26T04:22:37.165 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: 11 0 2024-03-26T04:22:37.165 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: 5 0 2024-03-26T04:22:37.165 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: 8 0 2024-03-26T04:22:37.165 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: 6 0 2024-03-26T04:22:37.166 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: 3 0 2024-03-26T04:22:37.166 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: 4 0 2024-03-26T04:22:37.166 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: 12 0 2024-03-26T04:22:37.166 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: 9 0 2024-03-26T04:22:37.166 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: 2 0 2024-03-26T04:22:37.166 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-03-26T04:22:37.166 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (42 ms) 2024-03-26T04:22:37.166 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: [----------] 3 tests from LibRadosListEC (1104 ms total) 2024-03-26T04:22:37.166 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: 2024-03-26T04:22:37.166 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-03-26T04:22:37.166 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-03-26T04:22:37.166 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3007 ms) 2024-03-26T04:22:37.166 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: [----------] 1 test from LibRadosListNP (3007 ms total) 2024-03-26T04:22:37.166 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: 2024-03-26T04:22:37.167 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: [----------] Global test environment tear-down 2024-03-26T04:22:37.167 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: [==========] 11 tests from 3 test suites ran. (168569 ms total) 2024-03-26T04:22:37.167 INFO:tasks.workunit.client.0.smithi016.stdout: api_list: [ PASSED ] 11 tests. 2024-03-26T04:22:38.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:37 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:22:38.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:37 smithi183 ceph-mon[33745]: pgmap v308: 340 pgs: 3 creating+peering, 35 unknown, 302 active+clean; 584 KiB data, 563 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:22:38.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:37 smithi183 ceph-mon[33745]: from='client.56537 ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi016-74374-3","pool2":"test-rados-api-smithi016-74374-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-03-26T04:22:38.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:37 smithi183 ceph-mon[33745]: osdmap e233: 8 total, 8 up, 8 in 2024-03-26T04:22:38.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:37 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/4195605130' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi016-74285-46"}]: dispatch 2024-03-26T04:22:38.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:37 smithi183 ceph-mon[33745]: from='client.49048 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi016-74285-46"}]: dispatch 2024-03-26T04:22:38.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:37 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:22:38.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:37 smithi016 ceph-mon[37215]: pgmap v308: 340 pgs: 3 creating+peering, 35 unknown, 302 active+clean; 584 KiB data, 563 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:22:38.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:37 smithi016 ceph-mon[37215]: from='client.56537 ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi016-74374-3","pool2":"test-rados-api-smithi016-74374-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-03-26T04:22:38.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:37 smithi016 ceph-mon[37215]: osdmap e233: 8 total, 8 up, 8 in 2024-03-26T04:22:38.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:37 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/4195605130' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi016-74285-46"}]: dispatch 2024-03-26T04:22:38.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:37 smithi016 ceph-mon[37215]: from='client.49048 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi016-74285-46"}]: dispatch 2024-03-26T04:22:38.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:37 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:22:38.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:37 smithi016 ceph-mon[29101]: pgmap v308: 340 pgs: 3 creating+peering, 35 unknown, 302 active+clean; 584 KiB data, 563 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:22:38.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:37 smithi016 ceph-mon[29101]: from='client.56537 ' entity='client.admin' cmd='[{"prefix":"osd pool rm","pool": "test-rados-api-smithi016-74374-3","pool2":"test-rados-api-smithi016-74374-3","yes_i_really_really_mean_it_not_faking": true}]': finished 2024-03-26T04:22:38.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:37 smithi016 ceph-mon[29101]: osdmap e233: 8 total, 8 up, 8 in 2024-03-26T04:22:38.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:37 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/4195605130' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi016-74285-46"}]: dispatch 2024-03-26T04:22:38.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:37 smithi016 ceph-mon[29101]: from='client.49048 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi016-74285-46"}]: dispatch 2024-03-26T04:22:39.161 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:22:38 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:22:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:22:39.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:39 smithi183 ceph-mon[33745]: from='client.49048 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi016-74285-46"}]': finished 2024-03-26T04:22:39.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:39 smithi183 ceph-mon[33745]: osdmap e234: 8 total, 8 up, 8 in 2024-03-26T04:22:39.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:39 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:39.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:39 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:39.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:39 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/4195605130' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi016-74285-46"}]: dispatch 2024-03-26T04:22:39.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:39 smithi183 ceph-mon[33745]: from='client.49048 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi016-74285-46"}]: dispatch 2024-03-26T04:22:39.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:39 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/266303643' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi016-74264-37"}]: dispatch 2024-03-26T04:22:39.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:39 smithi183 ceph-mon[33745]: from='client.49054 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi016-74264-37"}]: dispatch 2024-03-26T04:22:39.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:39 smithi016 ceph-mon[29101]: from='client.49048 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi016-74285-46"}]': finished 2024-03-26T04:22:39.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:39 smithi016 ceph-mon[29101]: osdmap e234: 8 total, 8 up, 8 in 2024-03-26T04:22:39.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:39 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:39.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:39 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:39.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:39 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/4195605130' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi016-74285-46"}]: dispatch 2024-03-26T04:22:39.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:39 smithi016 ceph-mon[29101]: from='client.49048 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi016-74285-46"}]: dispatch 2024-03-26T04:22:39.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:39 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/266303643' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi016-74264-37"}]: dispatch 2024-03-26T04:22:39.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:39 smithi016 ceph-mon[29101]: from='client.49054 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi016-74264-37"}]: dispatch 2024-03-26T04:22:39.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:39 smithi016 ceph-mon[37215]: from='client.49048 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi016-74285-46"}]': finished 2024-03-26T04:22:39.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:39 smithi016 ceph-mon[37215]: osdmap e234: 8 total, 8 up, 8 in 2024-03-26T04:22:39.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:39 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:39.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:39 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-29","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:39.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:39 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/4195605130' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi016-74285-46"}]: dispatch 2024-03-26T04:22:39.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:39 smithi016 ceph-mon[37215]: from='client.49048 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi016-74285-46"}]: dispatch 2024-03-26T04:22:39.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:39 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/266303643' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi016-74264-37"}]: dispatch 2024-03-26T04:22:39.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:39 smithi016 ceph-mon[37215]: from='client.49054 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi016-74264-37"}]: dispatch 2024-03-26T04:22:40.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:40 smithi183 ceph-mon[33745]: pgmap v311: 292 pgs: 32 unknown, 260 active+clean; 584 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:22:40.712 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:40 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:40.713 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:40 smithi183 ceph-mon[33745]: from='client.49048 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi016-74285-46"}]': finished 2024-03-26T04:22:40.713 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:40 smithi183 ceph-mon[33745]: from='client.49054 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi016-74264-37"}]': finished 2024-03-26T04:22:40.713 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:40 smithi183 ceph-mon[33745]: osdmap e235: 8 total, 8 up, 8 in 2024-03-26T04:22:40.713 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:40 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/266303643' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi016-74264-37"}]: dispatch 2024-03-26T04:22:40.713 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:40 smithi183 ceph-mon[33745]: from='client.49054 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi016-74264-37"}]: dispatch 2024-03-26T04:22:40.713 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:40 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/4194588887' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi016-74285-47"}]: dispatch 2024-03-26T04:22:40.713 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:40 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/4194588887' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi016-74285-47"}]: dispatch 2024-03-26T04:22:40.713 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:40 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/4194588887' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi016-74285-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:40.714 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:40 smithi016 ceph-mon[37215]: pgmap v311: 292 pgs: 32 unknown, 260 active+clean; 584 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:22:40.714 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:40 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:40.714 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:40 smithi016 ceph-mon[37215]: from='client.49048 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi016-74285-46"}]': finished 2024-03-26T04:22:40.714 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:40 smithi016 ceph-mon[37215]: from='client.49054 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi016-74264-37"}]': finished 2024-03-26T04:22:40.714 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:40 smithi016 ceph-mon[37215]: osdmap e235: 8 total, 8 up, 8 in 2024-03-26T04:22:40.715 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:40 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/266303643' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi016-74264-37"}]: dispatch 2024-03-26T04:22:40.715 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:40 smithi016 ceph-mon[37215]: from='client.49054 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi016-74264-37"}]: dispatch 2024-03-26T04:22:40.715 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:40 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/4194588887' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi016-74285-47"}]: dispatch 2024-03-26T04:22:40.715 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:40 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/4194588887' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi016-74285-47"}]: dispatch 2024-03-26T04:22:40.715 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:40 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/4194588887' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi016-74285-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:40.715 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:40 smithi016 ceph-mon[29101]: pgmap v311: 292 pgs: 32 unknown, 260 active+clean; 584 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:22:40.715 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:40 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-29","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:40.715 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:40 smithi016 ceph-mon[29101]: from='client.49048 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi016-74285-46"}]': finished 2024-03-26T04:22:40.715 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:40 smithi016 ceph-mon[29101]: from='client.49054 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFull_smithi016-74264-37"}]': finished 2024-03-26T04:22:40.715 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:40 smithi016 ceph-mon[29101]: osdmap e235: 8 total, 8 up, 8 in 2024-03-26T04:22:40.716 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:40 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/266303643' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi016-74264-37"}]: dispatch 2024-03-26T04:22:40.716 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:40 smithi016 ceph-mon[29101]: from='client.49054 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi016-74264-37"}]: dispatch 2024-03-26T04:22:40.716 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:40 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/4194588887' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi016-74285-47"}]: dispatch 2024-03-26T04:22:40.716 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:40 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/4194588887' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi016-74285-47"}]: dispatch 2024-03-26T04:22:40.716 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:40 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/4194588887' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi016-74285-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:41.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:41 smithi016 ceph-mon[29101]: from='client.49054 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi016-74264-37"}]': finished 2024-03-26T04:22:41.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:41 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/4194588887' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi016-74285-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:22:41.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:41 smithi016 ceph-mon[29101]: osdmap e236: 8 total, 8 up, 8 in 2024-03-26T04:22:41.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:41 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/4194588887' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi016-74285-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi016-74285-47"}]: dispatch 2024-03-26T04:22:41.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:41 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3655212285' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi016-74264-38"}]: dispatch 2024-03-26T04:22:41.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:41 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3655212285' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi016-74264-38"}]: dispatch 2024-03-26T04:22:41.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:41 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3655212285' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi016-74264-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:41.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:41 smithi016 ceph-mon[29101]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:22:41.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:41 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3655212285' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi016-74264-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:22:41.666 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:41 smithi016 ceph-mon[37215]: from='client.49054 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi016-74264-37"}]': finished 2024-03-26T04:22:41.667 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:41 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/4194588887' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi016-74285-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:22:41.667 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:41 smithi016 ceph-mon[37215]: osdmap e236: 8 total, 8 up, 8 in 2024-03-26T04:22:41.667 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:41 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/4194588887' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi016-74285-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi016-74285-47"}]: dispatch 2024-03-26T04:22:41.667 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:41 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3655212285' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi016-74264-38"}]: dispatch 2024-03-26T04:22:41.667 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:41 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3655212285' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi016-74264-38"}]: dispatch 2024-03-26T04:22:41.667 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:41 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3655212285' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi016-74264-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:41.667 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:41 smithi016 ceph-mon[37215]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:22:41.667 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:41 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3655212285' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi016-74264-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:22:41.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:41 smithi183 ceph-mon[33745]: from='client.49054 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFull_smithi016-74264-37"}]': finished 2024-03-26T04:22:41.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:41 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/4194588887' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleWritePP_smithi016-74285-47", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:22:41.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:41 smithi183 ceph-mon[33745]: osdmap e236: 8 total, 8 up, 8 in 2024-03-26T04:22:41.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:41 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/4194588887' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi016-74285-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi016-74285-47"}]: dispatch 2024-03-26T04:22:41.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:41 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3655212285' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi016-74264-38"}]: dispatch 2024-03-26T04:22:41.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:41 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3655212285' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi016-74264-38"}]: dispatch 2024-03-26T04:22:41.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:41 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3655212285' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi016-74264-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:41.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:41 smithi183 ceph-mon[33745]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:22:41.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:41 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3655212285' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStat_smithi016-74264-38", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:22:42.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:22:42 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:22:42.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:42 smithi183 ceph-mon[33745]: pgmap v314: 292 pgs: 32 unknown, 260 active+clean; 584 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:22:42.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:42 smithi183 ceph-mon[33745]: osdmap e237: 8 total, 8 up, 8 in 2024-03-26T04:22:42.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:42 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3655212285' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi016-74264-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi016-74264-38"}]: dispatch 2024-03-26T04:22:42.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:42 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:22:42.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:42 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:22:42.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:42 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/4194588887' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi016-74285-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi016-74285-47"}]': finished 2024-03-26T04:22:42.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:42 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-29", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:22:42.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:42 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-29"}]: dispatch 2024-03-26T04:22:42.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:42 smithi183 ceph-mon[33745]: osdmap e238: 8 total, 8 up, 8 in 2024-03-26T04:22:42.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:42 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-29"}]: dispatch 2024-03-26T04:22:42.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:42 smithi016 ceph-mon[29101]: pgmap v314: 292 pgs: 32 unknown, 260 active+clean; 584 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:22:42.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:42 smithi016 ceph-mon[29101]: osdmap e237: 8 total, 8 up, 8 in 2024-03-26T04:22:42.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:42 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3655212285' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi016-74264-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi016-74264-38"}]: dispatch 2024-03-26T04:22:42.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:42 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:22:42.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:42 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:22:42.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:42 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/4194588887' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi016-74285-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi016-74285-47"}]': finished 2024-03-26T04:22:42.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:42 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-29", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:22:42.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:42 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-29"}]: dispatch 2024-03-26T04:22:42.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:42 smithi016 ceph-mon[29101]: osdmap e238: 8 total, 8 up, 8 in 2024-03-26T04:22:42.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:42 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-29"}]: dispatch 2024-03-26T04:22:42.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:42 smithi016 ceph-mon[37215]: pgmap v314: 292 pgs: 32 unknown, 260 active+clean; 584 KiB data, 567 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:22:42.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:42 smithi016 ceph-mon[37215]: osdmap e237: 8 total, 8 up, 8 in 2024-03-26T04:22:42.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:42 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3655212285' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStat_smithi016-74264-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi016-74264-38"}]: dispatch 2024-03-26T04:22:42.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:42 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:22:42.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:42 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-29", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:22:42.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:42 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/4194588887' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritePP_smithi016-74285-47", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritePP_smithi016-74285-47"}]': finished 2024-03-26T04:22:42.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:42 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-29", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:22:42.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:42 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-29"}]: dispatch 2024-03-26T04:22:42.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:42 smithi016 ceph-mon[37215]: osdmap e238: 8 total, 8 up, 8 in 2024-03-26T04:22:42.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:42 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-29"}]: dispatch 2024-03-26T04:22:43.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:43 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:22:43.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:43 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3655212285' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi016-74264-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi016-74264-38"}]': finished 2024-03-26T04:22:43.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:43 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-29"}]': finished 2024-03-26T04:22:43.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:43 smithi016 ceph-mon[29101]: osdmap e239: 8 total, 8 up, 8 in 2024-03-26T04:22:43.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:43 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-29", "mode": "writeback"}]: dispatch 2024-03-26T04:22:43.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:43 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-29", "mode": "writeback"}]: dispatch 2024-03-26T04:22:43.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:43 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:22:43.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:43 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3655212285' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi016-74264-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi016-74264-38"}]': finished 2024-03-26T04:22:43.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:43 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-29"}]': finished 2024-03-26T04:22:43.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:43 smithi016 ceph-mon[37215]: osdmap e239: 8 total, 8 up, 8 in 2024-03-26T04:22:43.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:43 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-29", "mode": "writeback"}]: dispatch 2024-03-26T04:22:43.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:43 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-29", "mode": "writeback"}]: dispatch 2024-03-26T04:22:43.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:43 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:22:43.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:43 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3655212285' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStat_smithi016-74264-38", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStat_smithi016-74264-38"}]': finished 2024-03-26T04:22:43.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:43 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-29"}]': finished 2024-03-26T04:22:43.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:43 smithi183 ceph-mon[33745]: osdmap e239: 8 total, 8 up, 8 in 2024-03-26T04:22:43.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:43 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-29", "mode": "writeback"}]: dispatch 2024-03-26T04:22:43.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:43 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-29", "mode": "writeback"}]: dispatch 2024-03-26T04:22:44.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:44 smithi016 ceph-mon[29101]: pgmap v317: 300 pgs: 8 unknown, 292 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-26T04:22:44.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:44 smithi016 ceph-mon[29101]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:22:44.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:44 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-29", "mode": "writeback"}]': finished 2024-03-26T04:22:44.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:44 smithi016 ceph-mon[29101]: osdmap e240: 8 total, 8 up, 8 in 2024-03-26T04:22:44.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:44 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/4194588887' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi016-74285-47"}]: dispatch 2024-03-26T04:22:44.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:44 smithi016 ceph-mon[37215]: pgmap v317: 300 pgs: 8 unknown, 292 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-26T04:22:44.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:44 smithi016 ceph-mon[37215]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:22:44.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:44 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-29", "mode": "writeback"}]': finished 2024-03-26T04:22:44.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:44 smithi016 ceph-mon[37215]: osdmap e240: 8 total, 8 up, 8 in 2024-03-26T04:22:44.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:44 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/4194588887' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi016-74285-47"}]: dispatch 2024-03-26T04:22:44.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:44 smithi183 ceph-mon[33745]: pgmap v317: 300 pgs: 8 unknown, 292 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-26T04:22:44.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:44 smithi183 ceph-mon[33745]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:22:44.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:44 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-29", "mode": "writeback"}]': finished 2024-03-26T04:22:44.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:44 smithi183 ceph-mon[33745]: osdmap e240: 8 total, 8 up, 8 in 2024-03-26T04:22:44.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:44 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/4194588887' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi016-74285-47"}]: dispatch 2024-03-26T04:22:46.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:46 smithi016 ceph-mon[37215]: pgmap v320: 300 pgs: 8 unknown, 292 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-26T04:22:46.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:46 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/4194588887' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi016-74285-47"}]': finished 2024-03-26T04:22:46.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:46 smithi016 ceph-mon[37215]: osdmap e241: 8 total, 8 up, 8 in 2024-03-26T04:22:46.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:46 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/4194588887' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi016-74285-47"}]: dispatch 2024-03-26T04:22:46.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:46 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3655212285' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi016-74264-38"}]: dispatch 2024-03-26T04:22:46.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:46 smithi016 ceph-mon[29101]: pgmap v320: 300 pgs: 8 unknown, 292 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-26T04:22:46.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:46 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/4194588887' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi016-74285-47"}]': finished 2024-03-26T04:22:46.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:46 smithi016 ceph-mon[29101]: osdmap e241: 8 total, 8 up, 8 in 2024-03-26T04:22:46.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:46 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/4194588887' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi016-74285-47"}]: dispatch 2024-03-26T04:22:46.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:46 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3655212285' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi016-74264-38"}]: dispatch 2024-03-26T04:22:46.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:46 smithi183 ceph-mon[33745]: pgmap v320: 300 pgs: 8 unknown, 292 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-26T04:22:46.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:46 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/4194588887' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritePP_smithi016-74285-47"}]': finished 2024-03-26T04:22:46.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:46 smithi183 ceph-mon[33745]: osdmap e241: 8 total, 8 up, 8 in 2024-03-26T04:22:46.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:46 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/4194588887' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi016-74285-47"}]: dispatch 2024-03-26T04:22:46.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:46 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3655212285' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi016-74264-38"}]: dispatch 2024-03-26T04:22:47.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:47 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/4194588887' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi016-74285-47"}]': finished 2024-03-26T04:22:47.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:47 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3655212285' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi016-74264-38"}]': finished 2024-03-26T04:22:47.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:47 smithi016 ceph-mon[37215]: osdmap e242: 8 total, 8 up, 8 in 2024-03-26T04:22:47.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:47 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3655212285' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi016-74264-38"}]: dispatch 2024-03-26T04:22:47.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:47 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2460277117' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi016-74285-48"}]: dispatch 2024-03-26T04:22:47.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:47 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2460277117' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi016-74285-48"}]: dispatch 2024-03-26T04:22:47.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:47 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2460277117' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi016-74285-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:47.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:47 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:22:47.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:47 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:22:47.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:47 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3655212285' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi016-74264-38"}]': finished 2024-03-26T04:22:47.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:47 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2460277117' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi016-74285-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:22:47.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:47 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:22:47.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:47 smithi016 ceph-mon[37215]: osdmap e243: 8 total, 8 up, 8 in 2024-03-26T04:22:47.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:47 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/4194588887' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi016-74285-47"}]': finished 2024-03-26T04:22:47.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:47 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3655212285' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi016-74264-38"}]': finished 2024-03-26T04:22:47.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:47 smithi016 ceph-mon[29101]: osdmap e242: 8 total, 8 up, 8 in 2024-03-26T04:22:47.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:47 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3655212285' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi016-74264-38"}]: dispatch 2024-03-26T04:22:47.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:47 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2460277117' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi016-74285-48"}]: dispatch 2024-03-26T04:22:47.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:47 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2460277117' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi016-74285-48"}]: dispatch 2024-03-26T04:22:47.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:47 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2460277117' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi016-74285-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:47.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:47 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:22:47.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:47 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:22:47.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:47 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3655212285' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi016-74264-38"}]': finished 2024-03-26T04:22:47.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:47 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2460277117' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi016-74285-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:22:47.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:47 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:22:47.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:47 smithi016 ceph-mon[29101]: osdmap e243: 8 total, 8 up, 8 in 2024-03-26T04:22:47.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:47 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/4194588887' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritePP_smithi016-74285-47"}]': finished 2024-03-26T04:22:47.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:47 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3655212285' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStat_smithi016-74264-38"}]': finished 2024-03-26T04:22:47.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:47 smithi183 ceph-mon[33745]: osdmap e242: 8 total, 8 up, 8 in 2024-03-26T04:22:47.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:47 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3655212285' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi016-74264-38"}]: dispatch 2024-03-26T04:22:47.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:47 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2460277117' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi016-74285-48"}]: dispatch 2024-03-26T04:22:47.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:47 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2460277117' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi016-74285-48"}]: dispatch 2024-03-26T04:22:47.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:47 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2460277117' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi016-74285-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:47.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:47 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:22:47.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:47 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:22:47.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:47 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3655212285' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStat_smithi016-74264-38"}]': finished 2024-03-26T04:22:47.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:47 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2460277117' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-WaitForSafePP_smithi016-74285-48", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:22:47.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:47 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:22:47.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:47 smithi183 ceph-mon[33745]: osdmap e243: 8 total, 8 up, 8 in 2024-03-26T04:22:48.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:48 smithi016 ceph-mon[37215]: pgmap v323: 292 pgs: 292 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:22:48.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:48 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-29"}]: dispatch 2024-03-26T04:22:48.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:48 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2460277117' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi016-74285-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi016-74285-48"}]: dispatch 2024-03-26T04:22:48.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:48 smithi016 ceph-mon[37215]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:22:48.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:48 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-29"}]: dispatch 2024-03-26T04:22:48.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:48 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3211888405' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi016-74264-39"}]: dispatch 2024-03-26T04:22:48.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:48 smithi016 ceph-mon[37215]: from='client.56561 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi016-74264-39"}]: dispatch 2024-03-26T04:22:48.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:48 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3211888405' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi016-74264-39"}]: dispatch 2024-03-26T04:22:48.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:48 smithi016 ceph-mon[37215]: from='client.56561 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi016-74264-39"}]: dispatch 2024-03-26T04:22:48.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:48 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3211888405' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi016-74264-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:48.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:48 smithi016 ceph-mon[37215]: from='client.56561 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi016-74264-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:48.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:48 smithi016 ceph-mon[37215]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:22:48.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:48 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-29"}]': finished 2024-03-26T04:22:48.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:48 smithi016 ceph-mon[29101]: pgmap v323: 292 pgs: 292 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:22:48.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:48 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-29"}]: dispatch 2024-03-26T04:22:48.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:48 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2460277117' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi016-74285-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi016-74285-48"}]: dispatch 2024-03-26T04:22:48.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:48 smithi016 ceph-mon[29101]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:22:48.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:48 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-29"}]: dispatch 2024-03-26T04:22:48.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:48 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3211888405' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi016-74264-39"}]: dispatch 2024-03-26T04:22:48.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:48 smithi016 ceph-mon[29101]: from='client.56561 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi016-74264-39"}]: dispatch 2024-03-26T04:22:48.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:48 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3211888405' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi016-74264-39"}]: dispatch 2024-03-26T04:22:48.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:48 smithi016 ceph-mon[29101]: from='client.56561 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi016-74264-39"}]: dispatch 2024-03-26T04:22:48.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:48 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3211888405' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi016-74264-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:48.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:48 smithi016 ceph-mon[29101]: from='client.56561 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi016-74264-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:48.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:48 smithi016 ceph-mon[29101]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:22:48.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:48 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-29"}]': finished 2024-03-26T04:22:48.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:48 smithi183 ceph-mon[33745]: pgmap v323: 292 pgs: 292 active+clean; 4.6 MiB data, 580 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:22:48.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:48 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-29"}]: dispatch 2024-03-26T04:22:48.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:48 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2460277117' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi016-74285-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi016-74285-48"}]: dispatch 2024-03-26T04:22:48.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:48 smithi183 ceph-mon[33745]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:22:48.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:48 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-29"}]: dispatch 2024-03-26T04:22:48.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:48 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3211888405' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi016-74264-39"}]: dispatch 2024-03-26T04:22:48.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:48 smithi183 ceph-mon[33745]: from='client.56561 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi016-74264-39"}]: dispatch 2024-03-26T04:22:48.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:48 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3211888405' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi016-74264-39"}]: dispatch 2024-03-26T04:22:48.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:48 smithi183 ceph-mon[33745]: from='client.56561 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi016-74264-39"}]: dispatch 2024-03-26T04:22:48.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:48 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3211888405' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi016-74264-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:48.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:48 smithi183 ceph-mon[33745]: from='client.56561 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi016-74264-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:48.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:48 smithi183 ceph-mon[33745]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:22:48.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:48 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-29"}]': finished 2024-03-26T04:22:49.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:22:48 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:22:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:22:49.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:49 smithi016 ceph-mon[37215]: from='client.56561 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi016-74264-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:22:49.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:49 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3211888405' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi016-74264-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi016-74264-39"}]: dispatch 2024-03-26T04:22:49.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:49 smithi016 ceph-mon[37215]: osdmap e244: 8 total, 8 up, 8 in 2024-03-26T04:22:49.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:49 smithi016 ceph-mon[37215]: from='client.56561 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi016-74264-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi016-74264-39"}]: dispatch 2024-03-26T04:22:49.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:49 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2460277117' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi016-74285-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi016-74285-48"}]': finished 2024-03-26T04:22:49.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:49 smithi016 ceph-mon[29101]: from='client.56561 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi016-74264-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:22:49.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:49 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3211888405' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi016-74264-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi016-74264-39"}]: dispatch 2024-03-26T04:22:49.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:49 smithi016 ceph-mon[29101]: osdmap e244: 8 total, 8 up, 8 in 2024-03-26T04:22:49.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:49 smithi016 ceph-mon[29101]: from='client.56561 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi016-74264-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi016-74264-39"}]: dispatch 2024-03-26T04:22:49.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:49 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2460277117' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi016-74285-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi016-74285-48"}]': finished 2024-03-26T04:22:50.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:49 smithi183 ceph-mon[33745]: from='client.56561 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatNS_smithi016-74264-39", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:22:50.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:49 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3211888405' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi016-74264-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi016-74264-39"}]: dispatch 2024-03-26T04:22:50.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:49 smithi183 ceph-mon[33745]: osdmap e244: 8 total, 8 up, 8 in 2024-03-26T04:22:50.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:49 smithi183 ceph-mon[33745]: from='client.56561 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi016-74264-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi016-74264-39"}]: dispatch 2024-03-26T04:22:50.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:49 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2460277117' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "WaitForSafePP_smithi016-74285-48", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-WaitForSafePP_smithi016-74285-48"}]': finished 2024-03-26T04:22:50.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:50 smithi016 ceph-mon[37215]: pgmap v326: 292 pgs: 292 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 1024 KiB/s wr, 27 op/s 2024-03-26T04:22:50.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:50 smithi016 ceph-mon[37215]: osdmap e245: 8 total, 8 up, 8 in 2024-03-26T04:22:50.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:50 smithi016 ceph-mon[37215]: from='client.56561 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi016-74264-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi016-74264-39"}]': finished 2024-03-26T04:22:50.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:50 smithi016 ceph-mon[37215]: osdmap e246: 8 total, 8 up, 8 in 2024-03-26T04:22:50.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:50 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:50.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:50 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:50.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:50 smithi016 ceph-mon[29101]: pgmap v326: 292 pgs: 292 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 1024 KiB/s wr, 27 op/s 2024-03-26T04:22:50.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:50 smithi016 ceph-mon[29101]: osdmap e245: 8 total, 8 up, 8 in 2024-03-26T04:22:50.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:50 smithi016 ceph-mon[29101]: from='client.56561 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi016-74264-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi016-74264-39"}]': finished 2024-03-26T04:22:50.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:50 smithi016 ceph-mon[29101]: osdmap e246: 8 total, 8 up, 8 in 2024-03-26T04:22:50.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:50 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:50.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:50 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:51.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:50 smithi183 ceph-mon[33745]: pgmap v326: 292 pgs: 292 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 1024 KiB/s wr, 27 op/s 2024-03-26T04:22:51.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:50 smithi183 ceph-mon[33745]: osdmap e245: 8 total, 8 up, 8 in 2024-03-26T04:22:51.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:50 smithi183 ceph-mon[33745]: from='client.56561 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatNS_smithi016-74264-39", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatNS_smithi016-74264-39"}]': finished 2024-03-26T04:22:51.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:50 smithi183 ceph-mon[33745]: osdmap e246: 8 total, 8 up, 8 in 2024-03-26T04:22:51.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:50 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:51.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:50 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-31","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:52.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:52 smithi016 ceph-mon[37215]: pgmap v329: 308 pgs: 48 unknown, 260 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-26T04:22:52.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:52 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:52.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:52 smithi016 ceph-mon[37215]: osdmap e247: 8 total, 8 up, 8 in 2024-03-26T04:22:52.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:52 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2460277117' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi016-74285-48"}]: dispatch 2024-03-26T04:22:52.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:52 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:22:52.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:52 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:22:52.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:52 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:22:52.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:52 smithi016 ceph-mon[29101]: pgmap v329: 308 pgs: 48 unknown, 260 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-26T04:22:52.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:52 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:52.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:52 smithi016 ceph-mon[29101]: osdmap e247: 8 total, 8 up, 8 in 2024-03-26T04:22:52.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:52 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2460277117' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi016-74285-48"}]: dispatch 2024-03-26T04:22:52.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:52 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:22:52.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:52 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:22:52.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:52 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:22:52.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:22:52 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:22:52.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:52 smithi183 ceph-mon[33745]: pgmap v329: 308 pgs: 48 unknown, 260 active+clean; 4.6 MiB data, 577 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1024 KiB/s wr, 1 op/s 2024-03-26T04:22:52.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:52 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-31","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:22:52.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:52 smithi183 ceph-mon[33745]: osdmap e247: 8 total, 8 up, 8 in 2024-03-26T04:22:52.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:52 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2460277117' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi016-74285-48"}]: dispatch 2024-03-26T04:22:52.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:52 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:22:52.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:52 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-31", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:22:52.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:52 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:22:53.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:53 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2460277117' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi016-74285-48"}]': finished 2024-03-26T04:22:53.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:53 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-31", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:22:53.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:53 smithi016 ceph-mon[29101]: osdmap e248: 8 total, 8 up, 8 in 2024-03-26T04:22:53.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:53 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-31"}]: dispatch 2024-03-26T04:22:53.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:53 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2460277117' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi016-74285-48"}]: dispatch 2024-03-26T04:22:53.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:53 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-31"}]: dispatch 2024-03-26T04:22:53.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:53 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3211888405' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi016-74264-39"}]: dispatch 2024-03-26T04:22:53.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:53 smithi016 ceph-mon[29101]: from='client.56561 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi016-74264-39"}]: dispatch 2024-03-26T04:22:53.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:53 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:22:53.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:53 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2460277117' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi016-74285-48"}]': finished 2024-03-26T04:22:53.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:53 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2460277117' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi016-74285-48"}]': finished 2024-03-26T04:22:53.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:53 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-31", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:22:53.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:53 smithi016 ceph-mon[37215]: osdmap e248: 8 total, 8 up, 8 in 2024-03-26T04:22:53.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:53 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-31"}]: dispatch 2024-03-26T04:22:53.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:53 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2460277117' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi016-74285-48"}]: dispatch 2024-03-26T04:22:53.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:53 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-31"}]: dispatch 2024-03-26T04:22:53.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:53 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3211888405' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi016-74264-39"}]: dispatch 2024-03-26T04:22:53.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:53 smithi016 ceph-mon[37215]: from='client.56561 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi016-74264-39"}]: dispatch 2024-03-26T04:22:53.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:53 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:22:53.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:53 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2460277117' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi016-74285-48"}]': finished 2024-03-26T04:22:53.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:53 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2460277117' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-WaitForSafePP_smithi016-74285-48"}]': finished 2024-03-26T04:22:53.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:53 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-31", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:22:53.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:53 smithi183 ceph-mon[33745]: osdmap e248: 8 total, 8 up, 8 in 2024-03-26T04:22:53.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:53 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-31"}]: dispatch 2024-03-26T04:22:53.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:53 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2460277117' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi016-74285-48"}]: dispatch 2024-03-26T04:22:53.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:53 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-31"}]: dispatch 2024-03-26T04:22:53.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:53 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3211888405' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi016-74264-39"}]: dispatch 2024-03-26T04:22:53.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:53 smithi183 ceph-mon[33745]: from='client.56561 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi016-74264-39"}]: dispatch 2024-03-26T04:22:53.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:53 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:22:53.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:53 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2460277117' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"WaitForSafePP_smithi016-74285-48"}]': finished 2024-03-26T04:22:54.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:54 smithi016 ceph-mon[29101]: pgmap v332: 292 pgs: 292 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:22:54.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:54 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-31"}]': finished 2024-03-26T04:22:54.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:54 smithi016 ceph-mon[29101]: from='client.56561 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi016-74264-39"}]': finished 2024-03-26T04:22:54.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:54 smithi016 ceph-mon[29101]: osdmap e249: 8 total, 8 up, 8 in 2024-03-26T04:22:54.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:54 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3211888405' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi016-74264-39"}]: dispatch 2024-03-26T04:22:54.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:54 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-31", "mode": "writeback"}]: dispatch 2024-03-26T04:22:54.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:54 smithi016 ceph-mon[29101]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:22:54.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:54 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-31", "mode": "writeback"}]: dispatch 2024-03-26T04:22:54.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:54 smithi016 ceph-mon[29101]: from='client.56561 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi016-74264-39"}]: dispatch 2024-03-26T04:22:54.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:54 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/786375927' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi016-74285-49"}]: dispatch 2024-03-26T04:22:54.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:54 smithi016 ceph-mon[29101]: from='client.49090 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi016-74285-49"}]: dispatch 2024-03-26T04:22:54.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:54 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/786375927' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi016-74285-49"}]: dispatch 2024-03-26T04:22:54.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:54 smithi016 ceph-mon[29101]: from='client.49090 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi016-74285-49"}]: dispatch 2024-03-26T04:22:54.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:54 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/786375927' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi016-74285-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:54.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:54 smithi016 ceph-mon[29101]: from='client.49090 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi016-74285-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:54.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:54 smithi016 ceph-mon[29101]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:22:54.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:54 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-31", "mode": "writeback"}]': finished 2024-03-26T04:22:54.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:54 smithi016 ceph-mon[29101]: from='client.56561 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi016-74264-39"}]': finished 2024-03-26T04:22:54.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:54 smithi016 ceph-mon[29101]: from='client.49090 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi016-74285-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:22:54.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:54 smithi016 ceph-mon[37215]: pgmap v332: 292 pgs: 292 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:22:54.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:54 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-31"}]': finished 2024-03-26T04:22:54.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:54 smithi016 ceph-mon[37215]: from='client.56561 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi016-74264-39"}]': finished 2024-03-26T04:22:54.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:54 smithi016 ceph-mon[37215]: osdmap e249: 8 total, 8 up, 8 in 2024-03-26T04:22:54.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:54 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3211888405' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi016-74264-39"}]: dispatch 2024-03-26T04:22:54.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:54 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-31", "mode": "writeback"}]: dispatch 2024-03-26T04:22:54.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:54 smithi016 ceph-mon[37215]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:22:54.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:54 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-31", "mode": "writeback"}]: dispatch 2024-03-26T04:22:54.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:54 smithi016 ceph-mon[37215]: from='client.56561 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi016-74264-39"}]: dispatch 2024-03-26T04:22:54.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:54 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/786375927' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi016-74285-49"}]: dispatch 2024-03-26T04:22:54.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:54 smithi016 ceph-mon[37215]: from='client.49090 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi016-74285-49"}]: dispatch 2024-03-26T04:22:54.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:54 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/786375927' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi016-74285-49"}]: dispatch 2024-03-26T04:22:54.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:54 smithi016 ceph-mon[37215]: from='client.49090 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi016-74285-49"}]: dispatch 2024-03-26T04:22:54.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:54 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/786375927' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi016-74285-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:54.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:54 smithi016 ceph-mon[37215]: from='client.49090 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi016-74285-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:54.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:54 smithi016 ceph-mon[37215]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:22:54.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:54 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-31", "mode": "writeback"}]': finished 2024-03-26T04:22:54.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:54 smithi016 ceph-mon[37215]: from='client.56561 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi016-74264-39"}]': finished 2024-03-26T04:22:54.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:54 smithi016 ceph-mon[37215]: from='client.49090 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi016-74285-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:22:54.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:54 smithi183 ceph-mon[33745]: pgmap v332: 292 pgs: 292 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:22:54.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:54 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-31"}]': finished 2024-03-26T04:22:54.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:54 smithi183 ceph-mon[33745]: from='client.56561 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatNS_smithi016-74264-39"}]': finished 2024-03-26T04:22:54.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:54 smithi183 ceph-mon[33745]: osdmap e249: 8 total, 8 up, 8 in 2024-03-26T04:22:54.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:54 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3211888405' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi016-74264-39"}]: dispatch 2024-03-26T04:22:54.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:54 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-31", "mode": "writeback"}]: dispatch 2024-03-26T04:22:54.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:54 smithi183 ceph-mon[33745]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:22:54.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:54 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-31", "mode": "writeback"}]: dispatch 2024-03-26T04:22:54.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:54 smithi183 ceph-mon[33745]: from='client.56561 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi016-74264-39"}]: dispatch 2024-03-26T04:22:54.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:54 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/786375927' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi016-74285-49"}]: dispatch 2024-03-26T04:22:54.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:54 smithi183 ceph-mon[33745]: from='client.49090 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi016-74285-49"}]: dispatch 2024-03-26T04:22:54.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:54 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/786375927' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi016-74285-49"}]: dispatch 2024-03-26T04:22:54.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:54 smithi183 ceph-mon[33745]: from='client.49090 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi016-74285-49"}]: dispatch 2024-03-26T04:22:54.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:54 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/786375927' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi016-74285-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:54.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:54 smithi183 ceph-mon[33745]: from='client.49090 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi016-74285-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:54.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:54 smithi183 ceph-mon[33745]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:22:54.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:54 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-31", "mode": "writeback"}]': finished 2024-03-26T04:22:54.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:54 smithi183 ceph-mon[33745]: from='client.56561 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatNS_smithi016-74264-39"}]': finished 2024-03-26T04:22:54.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:54 smithi183 ceph-mon[33745]: from='client.49090 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP_smithi016-74285-49", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:22:55.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:55 smithi016 ceph-mon[37215]: osdmap e250: 8 total, 8 up, 8 in 2024-03-26T04:22:55.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:55 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/786375927' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi016-74285-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi016-74285-49"}]: dispatch 2024-03-26T04:22:55.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:55 smithi016 ceph-mon[37215]: from='client.49090 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi016-74285-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi016-74285-49"}]: dispatch 2024-03-26T04:22:55.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:55 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/905535448' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi016-74264-40"}]: dispatch 2024-03-26T04:22:55.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:55 smithi016 ceph-mon[37215]: from='client.49096 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi016-74264-40"}]: dispatch 2024-03-26T04:22:55.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:55 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/905535448' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi016-74264-40"}]: dispatch 2024-03-26T04:22:55.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:55 smithi016 ceph-mon[37215]: from='client.49096 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi016-74264-40"}]: dispatch 2024-03-26T04:22:55.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:55 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/905535448' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi016-74264-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:55.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:55 smithi016 ceph-mon[37215]: from='client.49096 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi016-74264-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:55.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:55 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:22:55.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:55 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:22:55.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:55 smithi016 ceph-mon[29101]: osdmap e250: 8 total, 8 up, 8 in 2024-03-26T04:22:55.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:55 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/786375927' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi016-74285-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi016-74285-49"}]: dispatch 2024-03-26T04:22:55.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:55 smithi016 ceph-mon[29101]: from='client.49090 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi016-74285-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi016-74285-49"}]: dispatch 2024-03-26T04:22:55.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:55 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/905535448' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi016-74264-40"}]: dispatch 2024-03-26T04:22:55.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:55 smithi016 ceph-mon[29101]: from='client.49096 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi016-74264-40"}]: dispatch 2024-03-26T04:22:55.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:55 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/905535448' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi016-74264-40"}]: dispatch 2024-03-26T04:22:55.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:55 smithi016 ceph-mon[29101]: from='client.49096 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi016-74264-40"}]: dispatch 2024-03-26T04:22:55.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:55 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/905535448' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi016-74264-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:55.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:55 smithi016 ceph-mon[29101]: from='client.49096 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi016-74264-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:55.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:55 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:22:55.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:55 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:22:55.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:55 smithi183 ceph-mon[33745]: osdmap e250: 8 total, 8 up, 8 in 2024-03-26T04:22:55.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:55 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/786375927' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi016-74285-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi016-74285-49"}]: dispatch 2024-03-26T04:22:55.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:55 smithi183 ceph-mon[33745]: from='client.49090 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi016-74285-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi016-74285-49"}]: dispatch 2024-03-26T04:22:55.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:55 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/905535448' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi016-74264-40"}]: dispatch 2024-03-26T04:22:55.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:55 smithi183 ceph-mon[33745]: from='client.49096 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi016-74264-40"}]: dispatch 2024-03-26T04:22:55.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:55 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/905535448' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi016-74264-40"}]: dispatch 2024-03-26T04:22:55.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:55 smithi183 ceph-mon[33745]: from='client.49096 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi016-74264-40"}]: dispatch 2024-03-26T04:22:55.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:55 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/905535448' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi016-74264-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:55.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:55 smithi183 ceph-mon[33745]: from='client.49096 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi016-74264-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:22:55.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:55 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:22:55.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:55 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:22:56.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:56 smithi016 ceph-mon[37215]: pgmap v335: 292 pgs: 292 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:22:56.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:56 smithi016 ceph-mon[37215]: from='client.49096 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi016-74264-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:22:56.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:56 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:22:56.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:56 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/905535448' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi016-74264-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi016-74264-40"}]: dispatch 2024-03-26T04:22:56.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:56 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-31"}]: dispatch 2024-03-26T04:22:56.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:56 smithi016 ceph-mon[37215]: osdmap e251: 8 total, 8 up, 8 in 2024-03-26T04:22:56.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:56 smithi016 ceph-mon[37215]: from='client.49096 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi016-74264-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi016-74264-40"}]: dispatch 2024-03-26T04:22:56.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:56 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-31"}]: dispatch 2024-03-26T04:22:56.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:56 smithi016 ceph-mon[29101]: pgmap v335: 292 pgs: 292 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:22:56.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:56 smithi016 ceph-mon[29101]: from='client.49096 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi016-74264-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:22:56.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:56 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:22:56.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:56 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/905535448' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi016-74264-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi016-74264-40"}]: dispatch 2024-03-26T04:22:56.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:56 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-31"}]: dispatch 2024-03-26T04:22:56.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:56 smithi016 ceph-mon[29101]: osdmap e251: 8 total, 8 up, 8 in 2024-03-26T04:22:56.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:56 smithi016 ceph-mon[29101]: from='client.49096 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi016-74264-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi016-74264-40"}]: dispatch 2024-03-26T04:22:56.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:56 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-31"}]: dispatch 2024-03-26T04:22:56.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:56 smithi183 ceph-mon[33745]: pgmap v335: 292 pgs: 292 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:22:56.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:56 smithi183 ceph-mon[33745]: from='client.49096 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemove_smithi016-74264-40", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:22:56.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:56 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:22:56.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:56 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/905535448' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi016-74264-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi016-74264-40"}]: dispatch 2024-03-26T04:22:56.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:56 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-31"}]: dispatch 2024-03-26T04:22:56.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:56 smithi183 ceph-mon[33745]: osdmap e251: 8 total, 8 up, 8 in 2024-03-26T04:22:56.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:56 smithi183 ceph-mon[33745]: from='client.49096 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemove_smithi016-74264-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi016-74264-40"}]: dispatch 2024-03-26T04:22:56.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:56 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-31"}]: dispatch 2024-03-26T04:22:57.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:57 smithi016 ceph-mon[37215]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:22:57.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:57 smithi016 ceph-mon[37215]: from='client.49090 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi016-74285-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi016-74285-49"}]': finished 2024-03-26T04:22:57.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:57 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-31"}]': finished 2024-03-26T04:22:57.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:57 smithi016 ceph-mon[37215]: osdmap e252: 8 total, 8 up, 8 in 2024-03-26T04:22:57.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:57 smithi016 ceph-mon[29101]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:22:57.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:57 smithi016 ceph-mon[29101]: from='client.49090 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi016-74285-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi016-74285-49"}]': finished 2024-03-26T04:22:57.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:57 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-31"}]': finished 2024-03-26T04:22:57.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:57 smithi016 ceph-mon[29101]: osdmap e252: 8 total, 8 up, 8 in 2024-03-26T04:22:57.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:57 smithi183 ceph-mon[33745]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:22:57.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:57 smithi183 ceph-mon[33745]: from='client.49090 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP_smithi016-74285-49", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP_smithi016-74285-49"}]': finished 2024-03-26T04:22:57.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:57 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-31"}]': finished 2024-03-26T04:22:57.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:57 smithi183 ceph-mon[33745]: osdmap e252: 8 total, 8 up, 8 in 2024-03-26T04:22:58.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:58 smithi016 ceph-mon[29101]: pgmap v338: 300 pgs: 2 creating+peering, 6 unknown, 292 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:22:58.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:58 smithi016 ceph-mon[29101]: from='client.49096 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi016-74264-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi016-74264-40"}]': finished 2024-03-26T04:22:58.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:58 smithi016 ceph-mon[29101]: osdmap e253: 8 total, 8 up, 8 in 2024-03-26T04:22:58.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:58 smithi016 ceph-mon[37215]: pgmap v338: 300 pgs: 2 creating+peering, 6 unknown, 292 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:22:58.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:58 smithi016 ceph-mon[37215]: from='client.49096 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi016-74264-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi016-74264-40"}]': finished 2024-03-26T04:22:58.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:58 smithi016 ceph-mon[37215]: osdmap e253: 8 total, 8 up, 8 in 2024-03-26T04:22:58.443 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:58 smithi183 ceph-mon[33745]: pgmap v338: 300 pgs: 2 creating+peering, 6 unknown, 292 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:22:58.443 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:58 smithi183 ceph-mon[33745]: from='client.49096 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemove_smithi016-74264-40", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemove_smithi016-74264-40"}]': finished 2024-03-26T04:22:58.443 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:58 smithi183 ceph-mon[33745]: osdmap e253: 8 total, 8 up, 8 in 2024-03-26T04:22:59.139 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:22:58 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:22:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:22:59.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:59 smithi016 ceph-mon[37215]: osdmap e254: 8 total, 8 up, 8 in 2024-03-26T04:22:59.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:59 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/786375927' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi016-74285-49"}]: dispatch 2024-03-26T04:22:59.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:59 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:59.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:59 smithi016 ceph-mon[37215]: from='client.49090 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi016-74285-49"}]: dispatch 2024-03-26T04:22:59.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:59 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:59.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:22:59 smithi016 ceph-mon[37215]: from='client.49090 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi016-74285-49"}]': finished 2024-03-26T04:22:59.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:59 smithi016 ceph-mon[29101]: osdmap e254: 8 total, 8 up, 8 in 2024-03-26T04:22:59.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:59 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/786375927' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi016-74285-49"}]: dispatch 2024-03-26T04:22:59.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:59 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:59.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:59 smithi016 ceph-mon[29101]: from='client.49090 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi016-74285-49"}]: dispatch 2024-03-26T04:22:59.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:59 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:59.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:22:59 smithi016 ceph-mon[29101]: from='client.49090 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi016-74285-49"}]': finished 2024-03-26T04:22:59.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:59 smithi183 ceph-mon[33745]: osdmap e254: 8 total, 8 up, 8 in 2024-03-26T04:22:59.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:59 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/786375927' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi016-74285-49"}]: dispatch 2024-03-26T04:22:59.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:59 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:59.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:59 smithi183 ceph-mon[33745]: from='client.49090 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi016-74285-49"}]: dispatch 2024-03-26T04:22:59.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:59 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-33","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:22:59.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:22:59 smithi183 ceph-mon[33745]: from='client.49090 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP_smithi016-74285-49"}]': finished 2024-03-26T04:23:00.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:00 smithi016 ceph-mon[29101]: pgmap v341: 300 pgs: 3 creating+peering, 37 unknown, 260 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:23:00.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:00 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:23:00.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:00 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/786375927' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi016-74285-49"}]: dispatch 2024-03-26T04:23:00.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:00 smithi016 ceph-mon[29101]: osdmap e255: 8 total, 8 up, 8 in 2024-03-26T04:23:00.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:00 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:23:00.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:00 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/905535448' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi016-74264-40"}]: dispatch 2024-03-26T04:23:00.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:00 smithi016 ceph-mon[29101]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:23:00.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:00 smithi016 ceph-mon[29101]: from='client.49090 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi016-74285-49"}]: dispatch 2024-03-26T04:23:00.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:00 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:23:00.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:00 smithi016 ceph-mon[29101]: from='client.49096 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi016-74264-40"}]: dispatch 2024-03-26T04:23:00.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:00 smithi016 ceph-mon[29101]: from='client.49090 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi016-74285-49"}]': finished 2024-03-26T04:23:00.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:00 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-33", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:23:00.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:00 smithi016 ceph-mon[29101]: from='client.49096 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi016-74264-40"}]': finished 2024-03-26T04:23:00.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:00 smithi016 ceph-mon[29101]: osdmap e256: 8 total, 8 up, 8 in 2024-03-26T04:23:00.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:00 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-33"}]: dispatch 2024-03-26T04:23:00.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:00 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/905535448' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi016-74264-40"}]: dispatch 2024-03-26T04:23:00.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:00 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-33"}]: dispatch 2024-03-26T04:23:00.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:00 smithi016 ceph-mon[29101]: from='client.49096 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi016-74264-40"}]: dispatch 2024-03-26T04:23:00.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:00 smithi016 ceph-mon[37215]: pgmap v341: 300 pgs: 3 creating+peering, 37 unknown, 260 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:23:00.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:00 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:23:00.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:00 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/786375927' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi016-74285-49"}]: dispatch 2024-03-26T04:23:00.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:00 smithi016 ceph-mon[37215]: osdmap e255: 8 total, 8 up, 8 in 2024-03-26T04:23:00.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:00 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:23:00.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:00 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/905535448' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi016-74264-40"}]: dispatch 2024-03-26T04:23:00.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:00 smithi016 ceph-mon[37215]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:23:00.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:00 smithi016 ceph-mon[37215]: from='client.49090 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi016-74285-49"}]: dispatch 2024-03-26T04:23:00.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:00 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:23:00.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:00 smithi016 ceph-mon[37215]: from='client.49096 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi016-74264-40"}]: dispatch 2024-03-26T04:23:00.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:00 smithi016 ceph-mon[37215]: from='client.49090 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi016-74285-49"}]': finished 2024-03-26T04:23:00.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:00 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-33", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:23:00.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:00 smithi016 ceph-mon[37215]: from='client.49096 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi016-74264-40"}]': finished 2024-03-26T04:23:00.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:00 smithi016 ceph-mon[37215]: osdmap e256: 8 total, 8 up, 8 in 2024-03-26T04:23:00.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:00 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-33"}]: dispatch 2024-03-26T04:23:00.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:00 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/905535448' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi016-74264-40"}]: dispatch 2024-03-26T04:23:00.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:00 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-33"}]: dispatch 2024-03-26T04:23:00.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:00 smithi016 ceph-mon[37215]: from='client.49096 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi016-74264-40"}]: dispatch 2024-03-26T04:23:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:00 smithi183 ceph-mon[33745]: pgmap v341: 300 pgs: 3 creating+peering, 37 unknown, 260 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:23:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:00 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-33","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:23:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:00 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/786375927' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi016-74285-49"}]: dispatch 2024-03-26T04:23:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:00 smithi183 ceph-mon[33745]: osdmap e255: 8 total, 8 up, 8 in 2024-03-26T04:23:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:00 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:23:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:00 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/905535448' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi016-74264-40"}]: dispatch 2024-03-26T04:23:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:00 smithi183 ceph-mon[33745]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:23:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:00 smithi183 ceph-mon[33745]: from='client.49090 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi016-74285-49"}]: dispatch 2024-03-26T04:23:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:00 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-33", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:23:00.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:00 smithi183 ceph-mon[33745]: from='client.49096 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi016-74264-40"}]: dispatch 2024-03-26T04:23:00.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:00 smithi183 ceph-mon[33745]: from='client.49090 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP_smithi016-74285-49"}]': finished 2024-03-26T04:23:00.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:00 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-33", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:23:00.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:00 smithi183 ceph-mon[33745]: from='client.49096 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemove_smithi016-74264-40"}]': finished 2024-03-26T04:23:00.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:00 smithi183 ceph-mon[33745]: osdmap e256: 8 total, 8 up, 8 in 2024-03-26T04:23:00.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:00 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-33"}]: dispatch 2024-03-26T04:23:00.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:00 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/905535448' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi016-74264-40"}]: dispatch 2024-03-26T04:23:00.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:00 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-33"}]: dispatch 2024-03-26T04:23:00.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:00 smithi183 ceph-mon[33745]: from='client.49096 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi016-74264-40"}]: dispatch 2024-03-26T04:23:01.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:01 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/595815812' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi016-74285-50"}]: dispatch 2024-03-26T04:23:01.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:01 smithi016 ceph-mon[29101]: from='client.56579 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi016-74285-50"}]: dispatch 2024-03-26T04:23:01.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:01 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/595815812' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi016-74285-50"}]: dispatch 2024-03-26T04:23:01.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:01 smithi016 ceph-mon[29101]: from='client.56579 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi016-74285-50"}]: dispatch 2024-03-26T04:23:01.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:01 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/595815812' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi016-74285-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:01.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:01 smithi016 ceph-mon[29101]: from='client.56579 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi016-74285-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:01.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:01 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-33"}]': finished 2024-03-26T04:23:01.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:01 smithi016 ceph-mon[29101]: from='client.49096 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi016-74264-40"}]': finished 2024-03-26T04:23:01.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:01 smithi016 ceph-mon[29101]: from='client.56579 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi016-74285-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:23:01.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:01 smithi016 ceph-mon[29101]: osdmap e257: 8 total, 8 up, 8 in 2024-03-26T04:23:01.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:01 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/595815812' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi016-74285-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi016-74285-50"}]: dispatch 2024-03-26T04:23:01.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:01 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-33", "mode": "writeback"}]: dispatch 2024-03-26T04:23:01.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:01 smithi016 ceph-mon[29101]: from='client.56579 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi016-74285-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi016-74285-50"}]: dispatch 2024-03-26T04:23:01.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:01 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-33", "mode": "writeback"}]: dispatch 2024-03-26T04:23:01.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:01 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/595815812' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi016-74285-50"}]: dispatch 2024-03-26T04:23:01.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:01 smithi016 ceph-mon[37215]: from='client.56579 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi016-74285-50"}]: dispatch 2024-03-26T04:23:01.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:01 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/595815812' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi016-74285-50"}]: dispatch 2024-03-26T04:23:01.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:01 smithi016 ceph-mon[37215]: from='client.56579 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi016-74285-50"}]: dispatch 2024-03-26T04:23:01.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:01 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/595815812' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi016-74285-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:01.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:01 smithi016 ceph-mon[37215]: from='client.56579 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi016-74285-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:01.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:01 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-33"}]': finished 2024-03-26T04:23:01.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:01 smithi016 ceph-mon[37215]: from='client.49096 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi016-74264-40"}]': finished 2024-03-26T04:23:01.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:01 smithi016 ceph-mon[37215]: from='client.56579 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi016-74285-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:23:01.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:01 smithi016 ceph-mon[37215]: osdmap e257: 8 total, 8 up, 8 in 2024-03-26T04:23:01.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:01 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/595815812' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi016-74285-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi016-74285-50"}]: dispatch 2024-03-26T04:23:01.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:01 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-33", "mode": "writeback"}]: dispatch 2024-03-26T04:23:01.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:01 smithi016 ceph-mon[37215]: from='client.56579 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi016-74285-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi016-74285-50"}]: dispatch 2024-03-26T04:23:01.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:01 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-33", "mode": "writeback"}]: dispatch 2024-03-26T04:23:01.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:01 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/595815812' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi016-74285-50"}]: dispatch 2024-03-26T04:23:01.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:01 smithi183 ceph-mon[33745]: from='client.56579 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi016-74285-50"}]: dispatch 2024-03-26T04:23:01.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:01 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/595815812' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi016-74285-50"}]: dispatch 2024-03-26T04:23:01.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:01 smithi183 ceph-mon[33745]: from='client.56579 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi016-74285-50"}]: dispatch 2024-03-26T04:23:01.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:01 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/595815812' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi016-74285-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:01.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:01 smithi183 ceph-mon[33745]: from='client.56579 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi016-74285-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:01.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:01 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-33"}]': finished 2024-03-26T04:23:01.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:01 smithi183 ceph-mon[33745]: from='client.49096 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemove_smithi016-74264-40"}]': finished 2024-03-26T04:23:01.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:01 smithi183 ceph-mon[33745]: from='client.56579 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripPP2_smithi016-74285-50", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:23:01.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:01 smithi183 ceph-mon[33745]: osdmap e257: 8 total, 8 up, 8 in 2024-03-26T04:23:01.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:01 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/595815812' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi016-74285-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi016-74285-50"}]: dispatch 2024-03-26T04:23:01.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:01 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-33", "mode": "writeback"}]: dispatch 2024-03-26T04:23:01.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:01 smithi183 ceph-mon[33745]: from='client.56579 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi016-74285-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi016-74285-50"}]: dispatch 2024-03-26T04:23:01.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:01 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-33", "mode": "writeback"}]: dispatch 2024-03-26T04:23:02.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:23:02 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:23:02.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:02 smithi183 ceph-mon[33745]: pgmap v344: 292 pgs: 32 unknown, 260 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:23:02.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:02 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1494070745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi016-74264-41"}]: dispatch 2024-03-26T04:23:02.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:02 smithi183 ceph-mon[33745]: from='client.56585 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi016-74264-41"}]: dispatch 2024-03-26T04:23:02.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:02 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1494070745' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi016-74264-41"}]: dispatch 2024-03-26T04:23:02.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:02 smithi183 ceph-mon[33745]: from='client.56585 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi016-74264-41"}]: dispatch 2024-03-26T04:23:02.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:02 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1494070745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi016-74264-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:02.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:02 smithi183 ceph-mon[33745]: from='client.56585 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi016-74264-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:02.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:02 smithi183 ceph-mon[33745]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:23:02.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:02 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-33", "mode": "writeback"}]': finished 2024-03-26T04:23:02.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:02 smithi016 ceph-mon[37215]: pgmap v344: 292 pgs: 32 unknown, 260 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:23:02.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:02 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1494070745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi016-74264-41"}]: dispatch 2024-03-26T04:23:02.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:02 smithi016 ceph-mon[37215]: from='client.56585 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi016-74264-41"}]: dispatch 2024-03-26T04:23:02.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:02 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1494070745' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi016-74264-41"}]: dispatch 2024-03-26T04:23:02.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:02 smithi016 ceph-mon[37215]: from='client.56585 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi016-74264-41"}]: dispatch 2024-03-26T04:23:02.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:02 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1494070745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi016-74264-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:02.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:02 smithi016 ceph-mon[37215]: from='client.56585 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi016-74264-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:02.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:02 smithi016 ceph-mon[37215]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:23:02.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:02 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-33", "mode": "writeback"}]': finished 2024-03-26T04:23:02.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:02 smithi016 ceph-mon[29101]: pgmap v344: 292 pgs: 32 unknown, 260 active+clean; 4.6 MiB data, 578 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:23:02.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:02 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1494070745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi016-74264-41"}]: dispatch 2024-03-26T04:23:02.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:02 smithi016 ceph-mon[29101]: from='client.56585 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi016-74264-41"}]: dispatch 2024-03-26T04:23:02.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:02 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1494070745' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi016-74264-41"}]: dispatch 2024-03-26T04:23:02.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:02 smithi016 ceph-mon[29101]: from='client.56585 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi016-74264-41"}]: dispatch 2024-03-26T04:23:02.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:02 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1494070745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi016-74264-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:02.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:02 smithi016 ceph-mon[29101]: from='client.56585 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi016-74264-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:02.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:02 smithi016 ceph-mon[29101]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:23:02.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:02 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-33", "mode": "writeback"}]': finished 2024-03-26T04:23:03.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:03 smithi016 ceph-mon[37215]: from='client.56585 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi016-74264-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:23:03.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:03 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1494070745' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi016-74264-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi016-74264-41"}]: dispatch 2024-03-26T04:23:03.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:03 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:23:03.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:03 smithi016 ceph-mon[37215]: osdmap e258: 8 total, 8 up, 8 in 2024-03-26T04:23:03.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:03 smithi016 ceph-mon[37215]: from='client.56585 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi016-74264-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi016-74264-41"}]: dispatch 2024-03-26T04:23:03.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:03 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:23:03.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:03 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:23:03.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:03 smithi016 ceph-mon[37215]: from='client.56579 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi016-74285-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi016-74285-50"}]': finished 2024-03-26T04:23:03.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:03 smithi016 ceph-mon[29101]: from='client.56585 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi016-74264-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:23:03.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:03 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1494070745' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi016-74264-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi016-74264-41"}]: dispatch 2024-03-26T04:23:03.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:03 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:23:03.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:03 smithi016 ceph-mon[29101]: osdmap e258: 8 total, 8 up, 8 in 2024-03-26T04:23:03.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:03 smithi016 ceph-mon[29101]: from='client.56585 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi016-74264-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi016-74264-41"}]: dispatch 2024-03-26T04:23:03.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:03 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:23:03.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:03 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:23:03.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:03 smithi016 ceph-mon[29101]: from='client.56579 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi016-74285-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi016-74285-50"}]': finished 2024-03-26T04:23:03.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:03 smithi183 ceph-mon[33745]: from='client.56585 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClass_smithi016-74264-41", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:23:03.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:03 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1494070745' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi016-74264-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi016-74264-41"}]: dispatch 2024-03-26T04:23:03.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:03 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:23:03.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:03 smithi183 ceph-mon[33745]: osdmap e258: 8 total, 8 up, 8 in 2024-03-26T04:23:03.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:03 smithi183 ceph-mon[33745]: from='client.56585 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi016-74264-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi016-74264-41"}]: dispatch 2024-03-26T04:23:03.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:03 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:23:03.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:03 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:23:03.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:03 smithi183 ceph-mon[33745]: from='client.56579 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripPP2_smithi016-74285-50", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripPP2_smithi016-74285-50"}]': finished 2024-03-26T04:23:04.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:04 smithi016 ceph-mon[37215]: pgmap v347: 292 pgs: 292 active+clean; 4.6 MiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:23:04.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:04 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:23:04.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:04 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-33"}]: dispatch 2024-03-26T04:23:04.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:04 smithi016 ceph-mon[37215]: osdmap e259: 8 total, 8 up, 8 in 2024-03-26T04:23:04.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:04 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-33"}]: dispatch 2024-03-26T04:23:04.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:04 smithi016 ceph-mon[37215]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:23:04.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:04 smithi016 ceph-mon[37215]: from='client.56585 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi016-74264-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi016-74264-41"}]': finished 2024-03-26T04:23:04.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:04 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-33"}]': finished 2024-03-26T04:23:04.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:04 smithi016 ceph-mon[37215]: osdmap e260: 8 total, 8 up, 8 in 2024-03-26T04:23:04.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:04 smithi016 ceph-mon[29101]: pgmap v347: 292 pgs: 292 active+clean; 4.6 MiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:23:04.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:04 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:23:04.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:04 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-33"}]: dispatch 2024-03-26T04:23:04.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:04 smithi016 ceph-mon[29101]: osdmap e259: 8 total, 8 up, 8 in 2024-03-26T04:23:04.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:04 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-33"}]: dispatch 2024-03-26T04:23:04.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:04 smithi016 ceph-mon[29101]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:23:04.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:04 smithi016 ceph-mon[29101]: from='client.56585 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi016-74264-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi016-74264-41"}]': finished 2024-03-26T04:23:04.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:04 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-33"}]': finished 2024-03-26T04:23:04.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:04 smithi016 ceph-mon[29101]: osdmap e260: 8 total, 8 up, 8 in 2024-03-26T04:23:04.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:04 smithi183 ceph-mon[33745]: pgmap v347: 292 pgs: 292 active+clean; 4.6 MiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:23:04.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:04 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:23:04.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:04 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-33"}]: dispatch 2024-03-26T04:23:04.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:04 smithi183 ceph-mon[33745]: osdmap e259: 8 total, 8 up, 8 in 2024-03-26T04:23:04.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:04 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-33"}]: dispatch 2024-03-26T04:23:04.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:04 smithi183 ceph-mon[33745]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:23:04.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:04 smithi183 ceph-mon[33745]: from='client.56585 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClass_smithi016-74264-41", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClass_smithi016-74264-41"}]': finished 2024-03-26T04:23:04.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:04 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-33"}]': finished 2024-03-26T04:23:04.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:04 smithi183 ceph-mon[33745]: osdmap e260: 8 total, 8 up, 8 in 2024-03-26T04:23:06.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:06 smithi016 ceph-mon[29101]: pgmap v350: 308 pgs: 16 unknown, 292 active+clean; 4.6 MiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:23:06.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:06 smithi016 ceph-mon[29101]: osdmap e261: 8 total, 8 up, 8 in 2024-03-26T04:23:06.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:06 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/595815812' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi016-74285-50"}]: dispatch 2024-03-26T04:23:06.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:06 smithi016 ceph-mon[29101]: from='client.56579 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi016-74285-50"}]: dispatch 2024-03-26T04:23:06.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:06 smithi016 ceph-mon[29101]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:23:06.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:06 smithi016 ceph-mon[37215]: pgmap v350: 308 pgs: 16 unknown, 292 active+clean; 4.6 MiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:23:06.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:06 smithi016 ceph-mon[37215]: osdmap e261: 8 total, 8 up, 8 in 2024-03-26T04:23:06.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:06 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/595815812' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi016-74285-50"}]: dispatch 2024-03-26T04:23:06.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:06 smithi016 ceph-mon[37215]: from='client.56579 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi016-74285-50"}]: dispatch 2024-03-26T04:23:06.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:06 smithi016 ceph-mon[37215]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:23:06.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:06 smithi183 ceph-mon[33745]: pgmap v350: 308 pgs: 16 unknown, 292 active+clean; 4.6 MiB data, 583 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:23:06.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:06 smithi183 ceph-mon[33745]: osdmap e261: 8 total, 8 up, 8 in 2024-03-26T04:23:06.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:06 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/595815812' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi016-74285-50"}]: dispatch 2024-03-26T04:23:06.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:06 smithi183 ceph-mon[33745]: from='client.56579 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi016-74285-50"}]: dispatch 2024-03-26T04:23:06.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:06 smithi183 ceph-mon[33745]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:23:07.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:07 smithi183 ceph-mon[33745]: from='client.56579 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi016-74285-50"}]': finished 2024-03-26T04:23:07.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:07 smithi183 ceph-mon[33745]: osdmap e262: 8 total, 8 up, 8 in 2024-03-26T04:23:07.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:07 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/595815812' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi016-74285-50"}]: dispatch 2024-03-26T04:23:07.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:07 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1494070745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi016-74264-41"}]: dispatch 2024-03-26T04:23:07.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:07 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:23:07.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:07 smithi183 ceph-mon[33745]: from='client.56579 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi016-74285-50"}]: dispatch 2024-03-26T04:23:07.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:07 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:23:07.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:07 smithi183 ceph-mon[33745]: from='client.56585 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi016-74264-41"}]: dispatch 2024-03-26T04:23:07.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:07 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:23:07.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:07 smithi183 ceph-mon[33745]: from='client.56579 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi016-74285-50"}]': finished 2024-03-26T04:23:07.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:07 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:23:07.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:07 smithi183 ceph-mon[33745]: from='client.56585 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi016-74264-41"}]': finished 2024-03-26T04:23:07.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:07 smithi183 ceph-mon[33745]: osdmap e263: 8 total, 8 up, 8 in 2024-03-26T04:23:07.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:07 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1494070745' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi016-74264-41"}]: dispatch 2024-03-26T04:23:07.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:07 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:23:07.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:07 smithi183 ceph-mon[33745]: from='client.56585 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi016-74264-41"}]: dispatch 2024-03-26T04:23:07.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:07 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:23:07.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:07 smithi016 ceph-mon[37215]: from='client.56579 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi016-74285-50"}]': finished 2024-03-26T04:23:07.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:07 smithi016 ceph-mon[37215]: osdmap e262: 8 total, 8 up, 8 in 2024-03-26T04:23:07.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:07 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/595815812' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi016-74285-50"}]: dispatch 2024-03-26T04:23:07.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:07 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1494070745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi016-74264-41"}]: dispatch 2024-03-26T04:23:07.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:07 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:23:07.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:07 smithi016 ceph-mon[37215]: from='client.56579 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi016-74285-50"}]: dispatch 2024-03-26T04:23:07.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:07 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:23:07.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:07 smithi016 ceph-mon[37215]: from='client.56585 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi016-74264-41"}]: dispatch 2024-03-26T04:23:07.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:07 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:23:07.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:07 smithi016 ceph-mon[37215]: from='client.56579 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi016-74285-50"}]': finished 2024-03-26T04:23:07.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:07 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:23:07.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:07 smithi016 ceph-mon[37215]: from='client.56585 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi016-74264-41"}]': finished 2024-03-26T04:23:07.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:07 smithi016 ceph-mon[37215]: osdmap e263: 8 total, 8 up, 8 in 2024-03-26T04:23:07.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:07 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1494070745' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi016-74264-41"}]: dispatch 2024-03-26T04:23:07.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:07 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:23:07.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:07 smithi016 ceph-mon[37215]: from='client.56585 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi016-74264-41"}]: dispatch 2024-03-26T04:23:07.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:07 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:23:07.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:07 smithi016 ceph-mon[29101]: from='client.56579 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripPP2_smithi016-74285-50"}]': finished 2024-03-26T04:23:07.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:07 smithi016 ceph-mon[29101]: osdmap e262: 8 total, 8 up, 8 in 2024-03-26T04:23:07.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:07 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/595815812' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi016-74285-50"}]: dispatch 2024-03-26T04:23:07.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:07 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1494070745' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi016-74264-41"}]: dispatch 2024-03-26T04:23:07.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:07 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:23:07.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:07 smithi016 ceph-mon[29101]: from='client.56579 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi016-74285-50"}]: dispatch 2024-03-26T04:23:07.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:07 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-35","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:23:07.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:07 smithi016 ceph-mon[29101]: from='client.56585 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi016-74264-41"}]: dispatch 2024-03-26T04:23:07.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:07 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:23:07.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:07 smithi016 ceph-mon[29101]: from='client.56579 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripPP2_smithi016-74285-50"}]': finished 2024-03-26T04:23:07.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:07 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-35","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:23:07.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:07 smithi016 ceph-mon[29101]: from='client.56585 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClass_smithi016-74264-41"}]': finished 2024-03-26T04:23:07.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:07 smithi016 ceph-mon[29101]: osdmap e263: 8 total, 8 up, 8 in 2024-03-26T04:23:07.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:07 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1494070745' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi016-74264-41"}]: dispatch 2024-03-26T04:23:07.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:07 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:23:07.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:07 smithi016 ceph-mon[29101]: from='client.56585 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi016-74264-41"}]: dispatch 2024-03-26T04:23:07.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:07 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-35", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:23:08.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:08 smithi183 ceph-mon[33745]: pgmap v353: 292 pgs: 7 creating+peering, 25 unknown, 260 active+clean; 4.6 MiB data, 583 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:23:08.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:08 smithi183 ceph-mon[33745]: from='client.56585 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi016-74264-41"}]': finished 2024-03-26T04:23:08.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:08 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-35", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:23:08.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:08 smithi183 ceph-mon[33745]: osdmap e264: 8 total, 8 up, 8 in 2024-03-26T04:23:08.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:08 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-35"}]: dispatch 2024-03-26T04:23:08.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:08 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2859378440' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi016-74285-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:23:08.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:08 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-35"}]: dispatch 2024-03-26T04:23:08.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:08 smithi183 ceph-mon[33745]: from='client.49114 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi016-74285-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:23:08.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:08 smithi016 ceph-mon[37215]: pgmap v353: 292 pgs: 7 creating+peering, 25 unknown, 260 active+clean; 4.6 MiB data, 583 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:23:08.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:08 smithi016 ceph-mon[37215]: from='client.56585 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi016-74264-41"}]': finished 2024-03-26T04:23:08.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:08 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-35", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:23:08.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:08 smithi016 ceph-mon[37215]: osdmap e264: 8 total, 8 up, 8 in 2024-03-26T04:23:08.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:08 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-35"}]: dispatch 2024-03-26T04:23:08.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:08 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2859378440' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi016-74285-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:23:08.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:08 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-35"}]: dispatch 2024-03-26T04:23:08.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:08 smithi016 ceph-mon[37215]: from='client.49114 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi016-74285-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:23:08.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:08 smithi016 ceph-mon[29101]: pgmap v353: 292 pgs: 7 creating+peering, 25 unknown, 260 active+clean; 4.6 MiB data, 583 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:23:08.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:08 smithi016 ceph-mon[29101]: from='client.56585 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClass_smithi016-74264-41"}]': finished 2024-03-26T04:23:08.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:08 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-35", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:23:08.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:08 smithi016 ceph-mon[29101]: osdmap e264: 8 total, 8 up, 8 in 2024-03-26T04:23:08.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:08 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-35"}]: dispatch 2024-03-26T04:23:08.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:08 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2859378440' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi016-74285-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:23:08.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:08 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-35"}]: dispatch 2024-03-26T04:23:08.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:08 smithi016 ceph-mon[29101]: from='client.49114 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi016-74285-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:23:09.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:23:08 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:23:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:23:09.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:09 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/534872064' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi016-74264-42"}]: dispatch 2024-03-26T04:23:09.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:09 smithi183 ceph-mon[33745]: from='client.56597 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi016-74264-42"}]: dispatch 2024-03-26T04:23:09.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:09 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/534872064' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi016-74264-42"}]: dispatch 2024-03-26T04:23:09.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:09 smithi183 ceph-mon[33745]: from='client.56597 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi016-74264-42"}]: dispatch 2024-03-26T04:23:09.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:09 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/534872064' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi016-74264-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:09.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:09 smithi183 ceph-mon[33745]: from='client.56597 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi016-74264-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:09.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:09 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-35"}]': finished 2024-03-26T04:23:09.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:09 smithi183 ceph-mon[33745]: from='client.49114 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi016-74285-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:23:09.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:09 smithi183 ceph-mon[33745]: from='client.56597 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi016-74264-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:23:09.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:09 smithi183 ceph-mon[33745]: osdmap e265: 8 total, 8 up, 8 in 2024-03-26T04:23:09.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:09 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/534872064' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi016-74264-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi016-74264-42"}]: dispatch 2024-03-26T04:23:09.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:09 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-35", "mode": "writeback"}]: dispatch 2024-03-26T04:23:09.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:09 smithi183 ceph-mon[33745]: from='client.56597 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi016-74264-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi016-74264-42"}]: dispatch 2024-03-26T04:23:09.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:09 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-35", "mode": "writeback"}]: dispatch 2024-03-26T04:23:09.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:09 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/534872064' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi016-74264-42"}]: dispatch 2024-03-26T04:23:09.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:09 smithi016 ceph-mon[37215]: from='client.56597 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi016-74264-42"}]: dispatch 2024-03-26T04:23:09.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:09 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/534872064' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi016-74264-42"}]: dispatch 2024-03-26T04:23:09.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:09 smithi016 ceph-mon[37215]: from='client.56597 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi016-74264-42"}]: dispatch 2024-03-26T04:23:09.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:09 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/534872064' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi016-74264-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:09.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:09 smithi016 ceph-mon[37215]: from='client.56597 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi016-74264-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:09.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:09 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-35"}]': finished 2024-03-26T04:23:09.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:09 smithi016 ceph-mon[37215]: from='client.49114 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi016-74285-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:23:09.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:09 smithi016 ceph-mon[37215]: from='client.56597 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi016-74264-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:23:09.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:09 smithi016 ceph-mon[37215]: osdmap e265: 8 total, 8 up, 8 in 2024-03-26T04:23:09.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:09 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/534872064' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi016-74264-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi016-74264-42"}]: dispatch 2024-03-26T04:23:09.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:09 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-35", "mode": "writeback"}]: dispatch 2024-03-26T04:23:09.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:09 smithi016 ceph-mon[37215]: from='client.56597 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi016-74264-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi016-74264-42"}]: dispatch 2024-03-26T04:23:09.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:09 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-35", "mode": "writeback"}]: dispatch 2024-03-26T04:23:09.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:09 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/534872064' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi016-74264-42"}]: dispatch 2024-03-26T04:23:09.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:09 smithi016 ceph-mon[29101]: from='client.56597 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi016-74264-42"}]: dispatch 2024-03-26T04:23:09.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:09 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/534872064' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi016-74264-42"}]: dispatch 2024-03-26T04:23:09.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:09 smithi016 ceph-mon[29101]: from='client.56597 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi016-74264-42"}]: dispatch 2024-03-26T04:23:09.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:09 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/534872064' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi016-74264-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:09.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:09 smithi016 ceph-mon[29101]: from='client.56597 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi016-74264-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:09.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:09 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-35"}]': finished 2024-03-26T04:23:09.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:09 smithi016 ceph-mon[29101]: from='client.49114 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripPP3_smithi016-74285-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:23:09.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:09 smithi016 ceph-mon[29101]: from='client.56597 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWrite_smithi016-74264-42", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:23:09.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:09 smithi016 ceph-mon[29101]: osdmap e265: 8 total, 8 up, 8 in 2024-03-26T04:23:09.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:09 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/534872064' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi016-74264-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi016-74264-42"}]: dispatch 2024-03-26T04:23:09.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:09 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-35", "mode": "writeback"}]: dispatch 2024-03-26T04:23:09.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:09 smithi016 ceph-mon[29101]: from='client.56597 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWrite_smithi016-74264-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi016-74264-42"}]: dispatch 2024-03-26T04:23:09.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:09 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-35", "mode": "writeback"}]: dispatch 2024-03-26T04:23:10.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:10 smithi016 ceph-mon[37215]: pgmap v356: 324 pgs: 32 unknown, 14 creating+peering, 278 active+clean; 4.6 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:23:10.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:10 smithi016 ceph-mon[37215]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:23:10.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:10 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-35", "mode": "writeback"}]': finished 2024-03-26T04:23:10.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:10 smithi016 ceph-mon[29101]: pgmap v356: 324 pgs: 32 unknown, 14 creating+peering, 278 active+clean; 4.6 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:23:10.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:10 smithi016 ceph-mon[29101]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:23:10.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:10 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-35", "mode": "writeback"}]': finished 2024-03-26T04:23:10.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:10 smithi183 ceph-mon[33745]: pgmap v356: 324 pgs: 32 unknown, 14 creating+peering, 278 active+clean; 4.6 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:23:10.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:10 smithi183 ceph-mon[33745]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:23:10.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:10 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-35", "mode": "writeback"}]': finished 2024-03-26T04:23:11.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:11 smithi183 ceph-mon[33745]: osdmap e266: 8 total, 8 up, 8 in 2024-03-26T04:23:11.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:11 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/468008116' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi016-74285-52"}]: dispatch 2024-03-26T04:23:11.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:11 smithi183 ceph-mon[33745]: from='client.56603 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi016-74285-52"}]: dispatch 2024-03-26T04:23:11.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:11 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/468008116' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi016-74285-52"}]: dispatch 2024-03-26T04:23:11.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:11 smithi183 ceph-mon[33745]: from='client.56603 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi016-74285-52"}]: dispatch 2024-03-26T04:23:11.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:11 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/468008116' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi016-74285-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:11.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:11 smithi183 ceph-mon[33745]: from='client.56603 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi016-74285-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:11.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:11 smithi183 ceph-mon[33745]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:23:11.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:11 smithi183 ceph-mon[33745]: from='client.56597 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi016-74264-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi016-74264-42"}]': finished 2024-03-26T04:23:11.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:11 smithi183 ceph-mon[33745]: from='client.56603 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi016-74285-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:23:11.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:11 smithi183 ceph-mon[33745]: osdmap e267: 8 total, 8 up, 8 in 2024-03-26T04:23:11.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:11 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/468008116' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi016-74285-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi016-74285-52"}]: dispatch 2024-03-26T04:23:11.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:11 smithi183 ceph-mon[33745]: from='client.56603 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi016-74285-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi016-74285-52"}]: dispatch 2024-03-26T04:23:11.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:11 smithi016 ceph-mon[37215]: osdmap e266: 8 total, 8 up, 8 in 2024-03-26T04:23:11.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:11 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/468008116' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi016-74285-52"}]: dispatch 2024-03-26T04:23:11.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:11 smithi016 ceph-mon[37215]: from='client.56603 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi016-74285-52"}]: dispatch 2024-03-26T04:23:11.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:11 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/468008116' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi016-74285-52"}]: dispatch 2024-03-26T04:23:11.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:11 smithi016 ceph-mon[37215]: from='client.56603 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi016-74285-52"}]: dispatch 2024-03-26T04:23:11.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:11 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/468008116' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi016-74285-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:11.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:11 smithi016 ceph-mon[37215]: from='client.56603 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi016-74285-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:11.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:11 smithi016 ceph-mon[37215]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:23:11.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:11 smithi016 ceph-mon[37215]: from='client.56597 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi016-74264-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi016-74264-42"}]': finished 2024-03-26T04:23:11.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:11 smithi016 ceph-mon[37215]: from='client.56603 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi016-74285-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:23:11.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:11 smithi016 ceph-mon[37215]: osdmap e267: 8 total, 8 up, 8 in 2024-03-26T04:23:11.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:11 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/468008116' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi016-74285-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi016-74285-52"}]: dispatch 2024-03-26T04:23:11.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:11 smithi016 ceph-mon[37215]: from='client.56603 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi016-74285-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi016-74285-52"}]: dispatch 2024-03-26T04:23:11.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:11 smithi016 ceph-mon[29101]: osdmap e266: 8 total, 8 up, 8 in 2024-03-26T04:23:11.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:11 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/468008116' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi016-74285-52"}]: dispatch 2024-03-26T04:23:11.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:11 smithi016 ceph-mon[29101]: from='client.56603 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi016-74285-52"}]: dispatch 2024-03-26T04:23:11.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:11 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/468008116' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi016-74285-52"}]: dispatch 2024-03-26T04:23:11.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:11 smithi016 ceph-mon[29101]: from='client.56603 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi016-74285-52"}]: dispatch 2024-03-26T04:23:11.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:11 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/468008116' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi016-74285-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:11.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:11 smithi016 ceph-mon[29101]: from='client.56603 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi016-74285-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:11.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:11 smithi016 ceph-mon[29101]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:23:11.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:11 smithi016 ceph-mon[29101]: from='client.56597 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWrite_smithi016-74264-42", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWrite_smithi016-74264-42"}]': finished 2024-03-26T04:23:11.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:11 smithi016 ceph-mon[29101]: from='client.56603 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripSparseReadPP_smithi016-74285-52", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:23:11.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:11 smithi016 ceph-mon[29101]: osdmap e267: 8 total, 8 up, 8 in 2024-03-26T04:23:11.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:11 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/468008116' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi016-74285-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi016-74285-52"}]: dispatch 2024-03-26T04:23:11.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:11 smithi016 ceph-mon[29101]: from='client.56603 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi016-74285-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi016-74285-52"}]: dispatch 2024-03-26T04:23:12.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:12 smithi183 ceph-mon[33745]: pgmap v359: 292 pgs: 14 creating+peering, 278 active+clean; 4.6 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:23:12.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:12 smithi183 ceph-mon[33745]: osdmap e268: 8 total, 8 up, 8 in 2024-03-26T04:23:12.542 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:23:12 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:23:12.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:12 smithi016 ceph-mon[37215]: pgmap v359: 292 pgs: 14 creating+peering, 278 active+clean; 4.6 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:23:12.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:12 smithi016 ceph-mon[37215]: osdmap e268: 8 total, 8 up, 8 in 2024-03-26T04:23:12.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:12 smithi016 ceph-mon[29101]: pgmap v359: 292 pgs: 14 creating+peering, 278 active+clean; 4.6 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:23:12.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:12 smithi016 ceph-mon[29101]: osdmap e268: 8 total, 8 up, 8 in 2024-03-26T04:23:13.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:13 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:23:13.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:13 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:23:13.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:13 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:23:13.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:13 smithi183 ceph-mon[33745]: from='client.56603 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi016-74285-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi016-74285-52"}]': finished 2024-03-26T04:23:13.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:13 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:23:13.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:13 smithi183 ceph-mon[33745]: osdmap e269: 8 total, 8 up, 8 in 2024-03-26T04:23:13.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:13 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/534872064' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi016-74264-42"}]: dispatch 2024-03-26T04:23:13.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:13 smithi183 ceph-mon[33745]: from='client.56597 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi016-74264-42"}]: dispatch 2024-03-26T04:23:13.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:13 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-35"}]: dispatch 2024-03-26T04:23:13.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:13 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-35"}]: dispatch 2024-03-26T04:23:13.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:13 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:23:13.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:13 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:23:13.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:13 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:23:13.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:13 smithi016 ceph-mon[37215]: from='client.56603 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi016-74285-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi016-74285-52"}]': finished 2024-03-26T04:23:13.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:13 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:23:13.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:13 smithi016 ceph-mon[37215]: osdmap e269: 8 total, 8 up, 8 in 2024-03-26T04:23:13.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:13 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/534872064' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi016-74264-42"}]: dispatch 2024-03-26T04:23:13.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:13 smithi016 ceph-mon[37215]: from='client.56597 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi016-74264-42"}]: dispatch 2024-03-26T04:23:13.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:13 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-35"}]: dispatch 2024-03-26T04:23:13.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:13 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-35"}]: dispatch 2024-03-26T04:23:13.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:13 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:23:13.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:13 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:23:13.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:13 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:23:13.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:13 smithi016 ceph-mon[29101]: from='client.56603 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripSparseReadPP_smithi016-74285-52", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripSparseReadPP_smithi016-74285-52"}]': finished 2024-03-26T04:23:13.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:13 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:23:13.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:13 smithi016 ceph-mon[29101]: osdmap e269: 8 total, 8 up, 8 in 2024-03-26T04:23:13.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:13 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/534872064' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi016-74264-42"}]: dispatch 2024-03-26T04:23:13.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:13 smithi016 ceph-mon[29101]: from='client.56597 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi016-74264-42"}]: dispatch 2024-03-26T04:23:13.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:13 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-35"}]: dispatch 2024-03-26T04:23:13.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:13 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-35"}]: dispatch 2024-03-26T04:23:14.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:14 smithi183 ceph-mon[33745]: pgmap v362: 300 pgs: 4 creating+activating, 4 creating+peering, 292 active+clean; 4.6 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:23:14.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:14 smithi183 ceph-mon[33745]: from='client.56597 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi016-74264-42"}]': finished 2024-03-26T04:23:14.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:14 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-35"}]': finished 2024-03-26T04:23:14.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:14 smithi183 ceph-mon[33745]: osdmap e270: 8 total, 8 up, 8 in 2024-03-26T04:23:14.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:14 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/534872064' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi016-74264-42"}]: dispatch 2024-03-26T04:23:14.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:14 smithi183 ceph-mon[33745]: from='client.56597 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi016-74264-42"}]: dispatch 2024-03-26T04:23:14.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:14 smithi016 ceph-mon[37215]: pgmap v362: 300 pgs: 4 creating+activating, 4 creating+peering, 292 active+clean; 4.6 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:23:14.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:14 smithi016 ceph-mon[37215]: from='client.56597 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi016-74264-42"}]': finished 2024-03-26T04:23:14.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:14 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-35"}]': finished 2024-03-26T04:23:14.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:14 smithi016 ceph-mon[37215]: osdmap e270: 8 total, 8 up, 8 in 2024-03-26T04:23:14.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:14 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/534872064' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi016-74264-42"}]: dispatch 2024-03-26T04:23:14.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:14 smithi016 ceph-mon[37215]: from='client.56597 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi016-74264-42"}]: dispatch 2024-03-26T04:23:14.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:14 smithi016 ceph-mon[29101]: pgmap v362: 300 pgs: 4 creating+activating, 4 creating+peering, 292 active+clean; 4.6 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:23:14.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:14 smithi016 ceph-mon[29101]: from='client.56597 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWrite_smithi016-74264-42"}]': finished 2024-03-26T04:23:14.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:14 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-35"}]': finished 2024-03-26T04:23:14.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:14 smithi016 ceph-mon[29101]: osdmap e270: 8 total, 8 up, 8 in 2024-03-26T04:23:14.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:14 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/534872064' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi016-74264-42"}]: dispatch 2024-03-26T04:23:14.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:14 smithi016 ceph-mon[29101]: from='client.56597 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi016-74264-42"}]: dispatch 2024-03-26T04:23:15.171 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (7010 ms) 2024-03-26T04:23:15.172 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-03-26T04:23:15.172 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (6962 ms) 2024-03-26T04:23:15.172 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-03-26T04:23:15.172 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (7063 ms) 2024-03-26T04:23:15.172 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-03-26T04:23:15.172 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (6949 ms) 2024-03-26T04:23:15.172 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-03-26T04:23:15.172 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (7029 ms) 2024-03-26T04:23:15.172 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-03-26T04:23:15.172 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (7029 ms) 2024-03-26T04:23:15.172 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-03-26T04:23:15.172 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (7023 ms) 2024-03-26T04:23:15.172 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-03-26T04:23:15.172 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (7023 ms) 2024-03-26T04:23:15.173 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-03-26T04:23:15.173 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (6924 ms) 2024-03-26T04:23:15.173 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-03-26T04:23:15.173 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (7030 ms) 2024-03-26T04:23:15.173 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-03-26T04:23:15.173 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (7020 ms) 2024-03-26T04:23:15.173 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-03-26T04:23:15.173 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (6937 ms) 2024-03-26T04:23:15.173 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-03-26T04:23:15.173 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (6996 ms) 2024-03-26T04:23:15.173 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-03-26T04:23:15.173 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (7032 ms) 2024-03-26T04:23:15.173 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-03-26T04:23:15.173 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7020 ms) 2024-03-26T04:23:15.173 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-03-26T04:23:15.174 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (7010 ms) 2024-03-26T04:23:15.174 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (112058 ms total) 2024-03-26T04:23:15.174 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: 2024-03-26T04:23:15.174 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [----------] Global test environment tear-down 2024-03-26T04:23:15.174 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (206639 ms total) 2024-03-26T04:23:15.174 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio: [ PASSED ] 42 tests. 2024-03-26T04:23:16.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:16 smithi183 ceph-mon[33745]: pgmap v365: 300 pgs: 8 unknown, 292 active+clean; 4.6 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:23:16.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:16 smithi183 ceph-mon[33745]: from='client.56597 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi016-74264-42"}]': finished 2024-03-26T04:23:16.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:16 smithi183 ceph-mon[33745]: osdmap e271: 8 total, 8 up, 8 in 2024-03-26T04:23:16.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:16 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/468008116' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi016-74285-52"}]: dispatch 2024-03-26T04:23:16.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:16 smithi183 ceph-mon[33745]: from='client.56603 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi016-74285-52"}]: dispatch 2024-03-26T04:23:16.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:16 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:23:16.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:16 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:23:16.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:16 smithi016 ceph-mon[37215]: pgmap v365: 300 pgs: 8 unknown, 292 active+clean; 4.6 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:23:16.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:16 smithi016 ceph-mon[37215]: from='client.56597 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi016-74264-42"}]': finished 2024-03-26T04:23:16.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:16 smithi016 ceph-mon[37215]: osdmap e271: 8 total, 8 up, 8 in 2024-03-26T04:23:16.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:16 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/468008116' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi016-74285-52"}]: dispatch 2024-03-26T04:23:16.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:16 smithi016 ceph-mon[37215]: from='client.56603 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi016-74285-52"}]: dispatch 2024-03-26T04:23:16.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:16 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:23:16.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:16 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:23:16.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:16 smithi016 ceph-mon[29101]: pgmap v365: 300 pgs: 8 unknown, 292 active+clean; 4.6 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:23:16.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:16 smithi016 ceph-mon[29101]: from='client.56597 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWrite_smithi016-74264-42"}]': finished 2024-03-26T04:23:16.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:16 smithi016 ceph-mon[29101]: osdmap e271: 8 total, 8 up, 8 in 2024-03-26T04:23:16.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:16 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/468008116' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi016-74285-52"}]: dispatch 2024-03-26T04:23:16.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:16 smithi016 ceph-mon[29101]: from='client.56603 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi016-74285-52"}]: dispatch 2024-03-26T04:23:16.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:16 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:23:16.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:16 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:23:17.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:17 smithi183 ceph-mon[33745]: from='client.56603 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi016-74285-52"}]': finished 2024-03-26T04:23:17.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:17 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:23:17.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:17 smithi183 ceph-mon[33745]: osdmap e272: 8 total, 8 up, 8 in 2024-03-26T04:23:17.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:17 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/468008116' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi016-74285-52"}]: dispatch 2024-03-26T04:23:17.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:17 smithi183 ceph-mon[33745]: from='client.56603 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi016-74285-52"}]: dispatch 2024-03-26T04:23:17.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:17 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-35"}]: dispatch 2024-03-26T04:23:17.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:17 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-35"}]: dispatch 2024-03-26T04:23:17.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:17 smithi183 ceph-mon[33745]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:23:17.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:17 smithi183 ceph-mon[33745]: from='client.56603 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi016-74285-52"}]': finished 2024-03-26T04:23:17.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:17 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-35"}]': finished 2024-03-26T04:23:17.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:17 smithi183 ceph-mon[33745]: osdmap e273: 8 total, 8 up, 8 in 2024-03-26T04:23:17.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:17 smithi016 ceph-mon[37215]: from='client.56603 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi016-74285-52"}]': finished 2024-03-26T04:23:17.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:17 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:23:17.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:17 smithi016 ceph-mon[37215]: osdmap e272: 8 total, 8 up, 8 in 2024-03-26T04:23:17.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:17 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/468008116' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi016-74285-52"}]: dispatch 2024-03-26T04:23:17.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:17 smithi016 ceph-mon[37215]: from='client.56603 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi016-74285-52"}]: dispatch 2024-03-26T04:23:17.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:17 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-35"}]: dispatch 2024-03-26T04:23:17.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:17 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-35"}]: dispatch 2024-03-26T04:23:17.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:17 smithi016 ceph-mon[37215]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:23:17.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:17 smithi016 ceph-mon[37215]: from='client.56603 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi016-74285-52"}]': finished 2024-03-26T04:23:17.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:17 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-35"}]': finished 2024-03-26T04:23:17.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:17 smithi016 ceph-mon[37215]: osdmap e273: 8 total, 8 up, 8 in 2024-03-26T04:23:17.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:17 smithi016 ceph-mon[29101]: from='client.56603 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripSparseReadPP_smithi016-74285-52"}]': finished 2024-03-26T04:23:17.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:17 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:23:17.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:17 smithi016 ceph-mon[29101]: osdmap e272: 8 total, 8 up, 8 in 2024-03-26T04:23:17.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:17 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/468008116' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi016-74285-52"}]: dispatch 2024-03-26T04:23:17.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:17 smithi016 ceph-mon[29101]: from='client.56603 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi016-74285-52"}]: dispatch 2024-03-26T04:23:17.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:17 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-35"}]: dispatch 2024-03-26T04:23:17.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:17 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-35"}]: dispatch 2024-03-26T04:23:17.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:17 smithi016 ceph-mon[29101]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:23:17.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:17 smithi016 ceph-mon[29101]: from='client.56603 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripSparseReadPP_smithi016-74285-52"}]': finished 2024-03-26T04:23:17.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:17 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-35"}]': finished 2024-03-26T04:23:17.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:17 smithi016 ceph-mon[29101]: osdmap e273: 8 total, 8 up, 8 in 2024-03-26T04:23:18.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:18 smithi183 ceph-mon[33745]: pgmap v368: 292 pgs: 292 active+clean; 4.6 MiB data, 585 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:23:18.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:18 smithi183 ceph-mon[33745]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:23:18.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:18 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2334802358' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi016-74285-53"}]: dispatch 2024-03-26T04:23:18.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:18 smithi183 ceph-mon[33745]: from='client.56609 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi016-74285-53"}]: dispatch 2024-03-26T04:23:18.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:18 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2334802358' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi016-74285-53"}]: dispatch 2024-03-26T04:23:18.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:18 smithi183 ceph-mon[33745]: from='client.56609 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi016-74285-53"}]: dispatch 2024-03-26T04:23:18.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:18 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2334802358' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi016-74285-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:18.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:18 smithi183 ceph-mon[33745]: from='client.56609 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi016-74285-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:18.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:18 smithi016 ceph-mon[37215]: pgmap v368: 292 pgs: 292 active+clean; 4.6 MiB data, 585 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:23:18.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:18 smithi016 ceph-mon[37215]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:23:18.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:18 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2334802358' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi016-74285-53"}]: dispatch 2024-03-26T04:23:18.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:18 smithi016 ceph-mon[37215]: from='client.56609 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi016-74285-53"}]: dispatch 2024-03-26T04:23:18.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:18 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2334802358' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi016-74285-53"}]: dispatch 2024-03-26T04:23:18.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:18 smithi016 ceph-mon[37215]: from='client.56609 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi016-74285-53"}]: dispatch 2024-03-26T04:23:18.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:18 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2334802358' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi016-74285-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:18.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:18 smithi016 ceph-mon[37215]: from='client.56609 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi016-74285-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:18.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:18 smithi016 ceph-mon[29101]: pgmap v368: 292 pgs: 292 active+clean; 4.6 MiB data, 585 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:23:18.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:18 smithi016 ceph-mon[29101]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:23:18.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:18 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2334802358' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi016-74285-53"}]: dispatch 2024-03-26T04:23:18.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:18 smithi016 ceph-mon[29101]: from='client.56609 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi016-74285-53"}]: dispatch 2024-03-26T04:23:18.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:18 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2334802358' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi016-74285-53"}]: dispatch 2024-03-26T04:23:18.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:18 smithi016 ceph-mon[29101]: from='client.56609 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi016-74285-53"}]: dispatch 2024-03-26T04:23:18.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:18 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2334802358' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi016-74285-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:18.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:18 smithi016 ceph-mon[29101]: from='client.56609 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi016-74285-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:19.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:23:18 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:23:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:23:19.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:19 smithi183 ceph-mon[33745]: from='client.56609 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi016-74285-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:23:19.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:19 smithi183 ceph-mon[33745]: osdmap e274: 8 total, 8 up, 8 in 2024-03-26T04:23:19.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:19 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2334802358' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi016-74285-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi016-74285-53"}]: dispatch 2024-03-26T04:23:19.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:19 smithi183 ceph-mon[33745]: from='client.56609 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi016-74285-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi016-74285-53"}]: dispatch 2024-03-26T04:23:19.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:19 smithi016 ceph-mon[37215]: from='client.56609 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi016-74285-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:23:19.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:19 smithi016 ceph-mon[37215]: osdmap e274: 8 total, 8 up, 8 in 2024-03-26T04:23:19.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:19 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2334802358' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi016-74285-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi016-74285-53"}]: dispatch 2024-03-26T04:23:19.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:19 smithi016 ceph-mon[37215]: from='client.56609 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi016-74285-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi016-74285-53"}]: dispatch 2024-03-26T04:23:19.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:19 smithi016 ceph-mon[29101]: from='client.56609 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripAppendPP_smithi016-74285-53", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:23:19.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:19 smithi016 ceph-mon[29101]: osdmap e274: 8 total, 8 up, 8 in 2024-03-26T04:23:19.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:19 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2334802358' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi016-74285-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi016-74285-53"}]: dispatch 2024-03-26T04:23:19.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:19 smithi016 ceph-mon[29101]: from='client.56609 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi016-74285-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi016-74285-53"}]: dispatch 2024-03-26T04:23:20.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:20 smithi183 ceph-mon[33745]: pgmap v371: 260 pgs: 260 active+clean; 4.6 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:23:20.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:20 smithi183 ceph-mon[33745]: osdmap e275: 8 total, 8 up, 8 in 2024-03-26T04:23:20.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:20 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:23:20.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:20 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:23:20.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:20 smithi016 ceph-mon[37215]: pgmap v371: 260 pgs: 260 active+clean; 4.6 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:23:20.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:20 smithi016 ceph-mon[37215]: osdmap e275: 8 total, 8 up, 8 in 2024-03-26T04:23:20.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:20 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:23:20.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:20 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:23:20.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:20 smithi016 ceph-mon[29101]: pgmap v371: 260 pgs: 260 active+clean; 4.6 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:23:20.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:20 smithi016 ceph-mon[29101]: osdmap e275: 8 total, 8 up, 8 in 2024-03-26T04:23:20.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:20 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:23:20.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:20 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-37","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:23:21.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:21 smithi183 ceph-mon[33745]: from='client.56609 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi016-74285-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi016-74285-53"}]': finished 2024-03-26T04:23:21.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:21 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:23:21.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:21 smithi183 ceph-mon[33745]: osdmap e276: 8 total, 8 up, 8 in 2024-03-26T04:23:21.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:21 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:23:21.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:21 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:23:21.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:21 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-37", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:23:21.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:21 smithi183 ceph-mon[33745]: osdmap e277: 8 total, 8 up, 8 in 2024-03-26T04:23:21.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:21 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-37"}]: dispatch 2024-03-26T04:23:21.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:21 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-37"}]: dispatch 2024-03-26T04:23:21.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:21 smithi016 ceph-mon[37215]: from='client.56609 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi016-74285-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi016-74285-53"}]': finished 2024-03-26T04:23:21.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:21 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:23:21.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:21 smithi016 ceph-mon[37215]: osdmap e276: 8 total, 8 up, 8 in 2024-03-26T04:23:21.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:21 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:23:21.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:21 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:23:21.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:21 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-37", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:23:21.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:21 smithi016 ceph-mon[37215]: osdmap e277: 8 total, 8 up, 8 in 2024-03-26T04:23:21.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:21 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-37"}]: dispatch 2024-03-26T04:23:21.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:21 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-37"}]: dispatch 2024-03-26T04:23:21.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:21 smithi016 ceph-mon[29101]: from='client.56609 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripAppendPP_smithi016-74285-53", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripAppendPP_smithi016-74285-53"}]': finished 2024-03-26T04:23:21.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:21 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-37","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:23:21.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:21 smithi016 ceph-mon[29101]: osdmap e276: 8 total, 8 up, 8 in 2024-03-26T04:23:21.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:21 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:23:21.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:21 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-37", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:23:21.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:21 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-37", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:23:21.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:21 smithi016 ceph-mon[29101]: osdmap e277: 8 total, 8 up, 8 in 2024-03-26T04:23:21.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:21 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-37"}]: dispatch 2024-03-26T04:23:21.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:21 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-37"}]: dispatch 2024-03-26T04:23:22.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:23:22 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:23:22.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:22 smithi183 ceph-mon[33745]: pgmap v374: 300 pgs: 40 unknown, 260 active+clean; 4.6 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:23:22.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:22 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:23:22.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:22 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-37"}]': finished 2024-03-26T04:23:22.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:22 smithi183 ceph-mon[33745]: osdmap e278: 8 total, 8 up, 8 in 2024-03-26T04:23:22.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:22 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-37", "mode": "writeback"}]: dispatch 2024-03-26T04:23:22.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:22 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2334802358' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi016-74285-53"}]: dispatch 2024-03-26T04:23:22.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:22 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-37", "mode": "writeback"}]: dispatch 2024-03-26T04:23:22.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:22 smithi183 ceph-mon[33745]: from='client.56609 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi016-74285-53"}]: dispatch 2024-03-26T04:23:22.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:22 smithi016 ceph-mon[29101]: pgmap v374: 300 pgs: 40 unknown, 260 active+clean; 4.6 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:23:22.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:22 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:23:22.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:22 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-37"}]': finished 2024-03-26T04:23:22.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:22 smithi016 ceph-mon[29101]: osdmap e278: 8 total, 8 up, 8 in 2024-03-26T04:23:22.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:22 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-37", "mode": "writeback"}]: dispatch 2024-03-26T04:23:22.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:22 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2334802358' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi016-74285-53"}]: dispatch 2024-03-26T04:23:22.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:22 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-37", "mode": "writeback"}]: dispatch 2024-03-26T04:23:22.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:22 smithi016 ceph-mon[29101]: from='client.56609 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi016-74285-53"}]: dispatch 2024-03-26T04:23:22.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:22 smithi016 ceph-mon[37215]: pgmap v374: 300 pgs: 40 unknown, 260 active+clean; 4.6 MiB data, 585 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:23:22.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:22 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:23:22.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:22 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-37"}]': finished 2024-03-26T04:23:22.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:22 smithi016 ceph-mon[37215]: osdmap e278: 8 total, 8 up, 8 in 2024-03-26T04:23:22.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:22 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-37", "mode": "writeback"}]: dispatch 2024-03-26T04:23:22.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:22 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2334802358' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi016-74285-53"}]: dispatch 2024-03-26T04:23:22.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:22 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-37", "mode": "writeback"}]: dispatch 2024-03-26T04:23:22.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:22 smithi016 ceph-mon[37215]: from='client.56609 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi016-74285-53"}]: dispatch 2024-03-26T04:23:23.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:23 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:23:23.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:23 smithi183 ceph-mon[33745]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:23:23.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:23 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-37", "mode": "writeback"}]': finished 2024-03-26T04:23:23.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:23 smithi183 ceph-mon[33745]: from='client.56609 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi016-74285-53"}]': finished 2024-03-26T04:23:23.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:23 smithi183 ceph-mon[33745]: osdmap e279: 8 total, 8 up, 8 in 2024-03-26T04:23:23.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:23 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2334802358' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi016-74285-53"}]: dispatch 2024-03-26T04:23:23.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:23 smithi183 ceph-mon[33745]: from='client.56609 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi016-74285-53"}]: dispatch 2024-03-26T04:23:23.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:23 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:23:23.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:23 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:23:23.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:23 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:23:23.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:23 smithi016 ceph-mon[37215]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:23:23.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:23 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-37", "mode": "writeback"}]': finished 2024-03-26T04:23:23.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:23 smithi016 ceph-mon[37215]: from='client.56609 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi016-74285-53"}]': finished 2024-03-26T04:23:23.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:23 smithi016 ceph-mon[37215]: osdmap e279: 8 total, 8 up, 8 in 2024-03-26T04:23:23.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:23 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2334802358' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi016-74285-53"}]: dispatch 2024-03-26T04:23:23.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:23 smithi016 ceph-mon[37215]: from='client.56609 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi016-74285-53"}]: dispatch 2024-03-26T04:23:23.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:23 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:23:23.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:23 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:23:23.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:23 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:23:23.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:23 smithi016 ceph-mon[29101]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:23:23.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:23 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-37", "mode": "writeback"}]': finished 2024-03-26T04:23:23.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:23 smithi016 ceph-mon[29101]: from='client.56609 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripAppendPP_smithi016-74285-53"}]': finished 2024-03-26T04:23:23.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:23 smithi016 ceph-mon[29101]: osdmap e279: 8 total, 8 up, 8 in 2024-03-26T04:23:23.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:23 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2334802358' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi016-74285-53"}]: dispatch 2024-03-26T04:23:23.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:23 smithi016 ceph-mon[29101]: from='client.56609 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi016-74285-53"}]: dispatch 2024-03-26T04:23:23.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:23 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:23:23.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:23 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:23:24.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:24 smithi183 ceph-mon[33745]: pgmap v377: 292 pgs: 292 active+clean; 4.6 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:23:24.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:24 smithi183 ceph-mon[33745]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:23:24.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:24 smithi183 ceph-mon[33745]: from='client.56609 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi016-74285-53"}]': finished 2024-03-26T04:23:24.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:24 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:23:24.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:24 smithi183 ceph-mon[33745]: osdmap e280: 8 total, 8 up, 8 in 2024-03-26T04:23:24.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:24 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-37"}]: dispatch 2024-03-26T04:23:24.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:24 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-37"}]: dispatch 2024-03-26T04:23:24.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:24 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2376552161' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi016-74285-54"}]: dispatch 2024-03-26T04:23:24.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:24 smithi183 ceph-mon[33745]: from='client.56615 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi016-74285-54"}]: dispatch 2024-03-26T04:23:24.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:24 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2376552161' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi016-74285-54"}]: dispatch 2024-03-26T04:23:24.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:24 smithi183 ceph-mon[33745]: from='client.56615 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi016-74285-54"}]: dispatch 2024-03-26T04:23:24.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:24 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2376552161' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi016-74285-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:24.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:24 smithi183 ceph-mon[33745]: from='client.56615 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi016-74285-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:24.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:24 smithi016 ceph-mon[37215]: pgmap v377: 292 pgs: 292 active+clean; 4.6 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:23:24.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:24 smithi016 ceph-mon[37215]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:23:24.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:24 smithi016 ceph-mon[37215]: from='client.56609 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi016-74285-53"}]': finished 2024-03-26T04:23:24.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:24 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:23:24.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:24 smithi016 ceph-mon[37215]: osdmap e280: 8 total, 8 up, 8 in 2024-03-26T04:23:24.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:24 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-37"}]: dispatch 2024-03-26T04:23:24.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:24 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-37"}]: dispatch 2024-03-26T04:23:24.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:24 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2376552161' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi016-74285-54"}]: dispatch 2024-03-26T04:23:24.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:24 smithi016 ceph-mon[37215]: from='client.56615 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi016-74285-54"}]: dispatch 2024-03-26T04:23:24.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:24 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2376552161' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi016-74285-54"}]: dispatch 2024-03-26T04:23:24.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:24 smithi016 ceph-mon[37215]: from='client.56615 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi016-74285-54"}]: dispatch 2024-03-26T04:23:24.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:24 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2376552161' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi016-74285-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:24.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:24 smithi016 ceph-mon[37215]: from='client.56615 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi016-74285-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:24.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:24 smithi016 ceph-mon[29101]: pgmap v377: 292 pgs: 292 active+clean; 4.6 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:23:24.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:24 smithi016 ceph-mon[29101]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:23:24.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:24 smithi016 ceph-mon[29101]: from='client.56609 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripAppendPP_smithi016-74285-53"}]': finished 2024-03-26T04:23:24.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:24 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:23:24.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:24 smithi016 ceph-mon[29101]: osdmap e280: 8 total, 8 up, 8 in 2024-03-26T04:23:24.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:24 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-37"}]: dispatch 2024-03-26T04:23:24.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:24 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-37"}]: dispatch 2024-03-26T04:23:24.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:24 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2376552161' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi016-74285-54"}]: dispatch 2024-03-26T04:23:24.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:24 smithi016 ceph-mon[29101]: from='client.56615 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi016-74285-54"}]: dispatch 2024-03-26T04:23:24.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:24 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2376552161' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi016-74285-54"}]: dispatch 2024-03-26T04:23:24.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:24 smithi016 ceph-mon[29101]: from='client.56615 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi016-74285-54"}]: dispatch 2024-03-26T04:23:24.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:24 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2376552161' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi016-74285-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:24.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:24 smithi016 ceph-mon[29101]: from='client.56615 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi016-74285-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:25.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:25 smithi183 ceph-mon[33745]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:23:25.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:25 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-37"}]': finished 2024-03-26T04:23:25.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:25 smithi183 ceph-mon[33745]: from='client.56615 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi016-74285-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:23:25.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:25 smithi183 ceph-mon[33745]: osdmap e281: 8 total, 8 up, 8 in 2024-03-26T04:23:25.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:25 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2376552161' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi016-74285-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi016-74285-54"}]: dispatch 2024-03-26T04:23:25.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:25 smithi183 ceph-mon[33745]: from='client.56615 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi016-74285-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi016-74285-54"}]: dispatch 2024-03-26T04:23:25.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:25 smithi016 ceph-mon[37215]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:23:25.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:25 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-37"}]': finished 2024-03-26T04:23:25.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:25 smithi016 ceph-mon[37215]: from='client.56615 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi016-74285-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:23:25.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:25 smithi016 ceph-mon[37215]: osdmap e281: 8 total, 8 up, 8 in 2024-03-26T04:23:25.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:25 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2376552161' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi016-74285-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi016-74285-54"}]: dispatch 2024-03-26T04:23:25.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:25 smithi016 ceph-mon[37215]: from='client.56615 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi016-74285-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi016-74285-54"}]: dispatch 2024-03-26T04:23:25.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:25 smithi016 ceph-mon[29101]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:23:25.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:25 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-37"}]': finished 2024-03-26T04:23:25.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:25 smithi016 ceph-mon[29101]: from='client.56615 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsCompletePP_smithi016-74285-54", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:23:25.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:25 smithi016 ceph-mon[29101]: osdmap e281: 8 total, 8 up, 8 in 2024-03-26T04:23:25.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:25 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2376552161' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi016-74285-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi016-74285-54"}]: dispatch 2024-03-26T04:23:25.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:25 smithi016 ceph-mon[29101]: from='client.56615 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi016-74285-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi016-74285-54"}]: dispatch 2024-03-26T04:23:26.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:26 smithi016 ceph-mon[37215]: pgmap v380: 292 pgs: 292 active+clean; 4.6 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:23:26.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:26 smithi016 ceph-mon[29101]: pgmap v380: 292 pgs: 292 active+clean; 4.6 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:23:26.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:26 smithi183 ceph-mon[33745]: pgmap v380: 292 pgs: 292 active+clean; 4.6 MiB data, 594 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:23:27.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:27 smithi016 ceph-mon[29101]: osdmap e282: 8 total, 8 up, 8 in 2024-03-26T04:23:27.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:27 smithi016 ceph-mon[29101]: from='client.56615 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi016-74285-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi016-74285-54"}]': finished 2024-03-26T04:23:27.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:27 smithi016 ceph-mon[29101]: osdmap e283: 8 total, 8 up, 8 in 2024-03-26T04:23:27.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:27 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:23:27.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:27 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:23:27.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:27 smithi016 ceph-mon[37215]: osdmap e282: 8 total, 8 up, 8 in 2024-03-26T04:23:27.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:27 smithi016 ceph-mon[37215]: from='client.56615 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi016-74285-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi016-74285-54"}]': finished 2024-03-26T04:23:27.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:27 smithi016 ceph-mon[37215]: osdmap e283: 8 total, 8 up, 8 in 2024-03-26T04:23:27.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:27 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:23:27.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:27 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:23:27.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:27 smithi183 ceph-mon[33745]: osdmap e282: 8 total, 8 up, 8 in 2024-03-26T04:23:27.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:27 smithi183 ceph-mon[33745]: from='client.56615 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsCompletePP_smithi016-74285-54", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsCompletePP_smithi016-74285-54"}]': finished 2024-03-26T04:23:27.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:27 smithi183 ceph-mon[33745]: osdmap e283: 8 total, 8 up, 8 in 2024-03-26T04:23:27.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:27 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:23:27.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:27 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-39","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:23:28.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:28 smithi016 ceph-mon[29101]: pgmap v383: 260 pgs: 260 active+clean; 4.6 MiB data, 594 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:23:28.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:28 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:23:28.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:28 smithi016 ceph-mon[29101]: osdmap e284: 8 total, 8 up, 8 in 2024-03-26T04:23:28.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:28 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:23:28.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:28 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:23:28.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:28 smithi016 ceph-mon[37215]: pgmap v383: 260 pgs: 260 active+clean; 4.6 MiB data, 594 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:23:28.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:28 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:23:28.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:28 smithi016 ceph-mon[37215]: osdmap e284: 8 total, 8 up, 8 in 2024-03-26T04:23:28.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:28 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:23:28.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:28 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:23:28.693 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:28 smithi183 ceph-mon[33745]: pgmap v383: 260 pgs: 260 active+clean; 4.6 MiB data, 594 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:23:28.693 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:28 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-39","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:23:28.693 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:28 smithi183 ceph-mon[33745]: osdmap e284: 8 total, 8 up, 8 in 2024-03-26T04:23:28.693 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:28 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:23:28.693 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:28 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-39", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:23:29.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:23:28 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:23:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:23:30.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:30 smithi016 ceph-mon[37215]: pgmap v386: 300 pgs: 14 creating+peering, 26 unknown, 260 active+clean; 4.6 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T04:23:30.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:30 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-39", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:23:30.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:30 smithi016 ceph-mon[37215]: osdmap e285: 8 total, 8 up, 8 in 2024-03-26T04:23:30.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:30 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-39"}]: dispatch 2024-03-26T04:23:30.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:30 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-39"}]: dispatch 2024-03-26T04:23:30.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:30 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2376552161' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi016-74285-54"}]: dispatch 2024-03-26T04:23:30.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:30 smithi016 ceph-mon[37215]: from='client.56615 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi016-74285-54"}]: dispatch 2024-03-26T04:23:30.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:30 smithi016 ceph-mon[37215]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:23:30.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:30 smithi016 ceph-mon[29101]: pgmap v386: 300 pgs: 14 creating+peering, 26 unknown, 260 active+clean; 4.6 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T04:23:30.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:30 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-39", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:23:30.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:30 smithi016 ceph-mon[29101]: osdmap e285: 8 total, 8 up, 8 in 2024-03-26T04:23:30.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:30 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-39"}]: dispatch 2024-03-26T04:23:30.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:30 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-39"}]: dispatch 2024-03-26T04:23:30.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:30 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2376552161' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi016-74285-54"}]: dispatch 2024-03-26T04:23:30.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:30 smithi016 ceph-mon[29101]: from='client.56615 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi016-74285-54"}]: dispatch 2024-03-26T04:23:30.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:30 smithi016 ceph-mon[29101]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:23:30.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:30 smithi183 ceph-mon[33745]: pgmap v386: 300 pgs: 14 creating+peering, 26 unknown, 260 active+clean; 4.6 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T04:23:30.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:30 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-39", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:23:30.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:30 smithi183 ceph-mon[33745]: osdmap e285: 8 total, 8 up, 8 in 2024-03-26T04:23:30.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:30 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-39"}]: dispatch 2024-03-26T04:23:30.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:30 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-39"}]: dispatch 2024-03-26T04:23:30.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:30 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2376552161' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi016-74285-54"}]: dispatch 2024-03-26T04:23:30.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:30 smithi183 ceph-mon[33745]: from='client.56615 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi016-74285-54"}]: dispatch 2024-03-26T04:23:30.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:30 smithi183 ceph-mon[33745]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:23:31.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:31 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-39"}]': finished 2024-03-26T04:23:31.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:31 smithi016 ceph-mon[37215]: from='client.56615 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi016-74285-54"}]': finished 2024-03-26T04:23:31.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:31 smithi016 ceph-mon[37215]: osdmap e286: 8 total, 8 up, 8 in 2024-03-26T04:23:31.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:31 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2376552161' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi016-74285-54"}]: dispatch 2024-03-26T04:23:31.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:31 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-39", "mode": "writeback"}]: dispatch 2024-03-26T04:23:31.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:31 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-39", "mode": "writeback"}]: dispatch 2024-03-26T04:23:31.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:31 smithi016 ceph-mon[37215]: from='client.56615 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi016-74285-54"}]: dispatch 2024-03-26T04:23:31.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:31 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:23:31.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:31 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-39"}]': finished 2024-03-26T04:23:31.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:31 smithi016 ceph-mon[29101]: from='client.56615 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi016-74285-54"}]': finished 2024-03-26T04:23:31.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:31 smithi016 ceph-mon[29101]: osdmap e286: 8 total, 8 up, 8 in 2024-03-26T04:23:31.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:31 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2376552161' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi016-74285-54"}]: dispatch 2024-03-26T04:23:31.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:31 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-39", "mode": "writeback"}]: dispatch 2024-03-26T04:23:31.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:31 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-39", "mode": "writeback"}]: dispatch 2024-03-26T04:23:31.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:31 smithi016 ceph-mon[29101]: from='client.56615 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi016-74285-54"}]: dispatch 2024-03-26T04:23:31.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:31 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:23:31.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:31 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-39"}]': finished 2024-03-26T04:23:31.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:31 smithi183 ceph-mon[33745]: from='client.56615 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsCompletePP_smithi016-74285-54"}]': finished 2024-03-26T04:23:31.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:31 smithi183 ceph-mon[33745]: osdmap e286: 8 total, 8 up, 8 in 2024-03-26T04:23:31.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:31 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2376552161' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi016-74285-54"}]: dispatch 2024-03-26T04:23:31.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:31 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-39", "mode": "writeback"}]: dispatch 2024-03-26T04:23:31.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:31 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-39", "mode": "writeback"}]: dispatch 2024-03-26T04:23:31.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:31 smithi183 ceph-mon[33745]: from='client.56615 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi016-74285-54"}]: dispatch 2024-03-26T04:23:31.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:31 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:23:32.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:32 smithi016 ceph-mon[37215]: pgmap v389: 292 pgs: 11 creating+peering, 21 unknown, 260 active+clean; 4.6 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T04:23:32.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:32 smithi016 ceph-mon[37215]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:23:32.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:32 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-39", "mode": "writeback"}]': finished 2024-03-26T04:23:32.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:32 smithi016 ceph-mon[37215]: from='client.56615 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi016-74285-54"}]': finished 2024-03-26T04:23:32.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:32 smithi016 ceph-mon[37215]: osdmap e287: 8 total, 8 up, 8 in 2024-03-26T04:23:32.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:32 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/535223606' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi016-74285-55"}]: dispatch 2024-03-26T04:23:32.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:32 smithi016 ceph-mon[37215]: from='client.49144 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi016-74285-55"}]: dispatch 2024-03-26T04:23:32.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:32 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/535223606' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi016-74285-55"}]: dispatch 2024-03-26T04:23:32.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:32 smithi016 ceph-mon[37215]: from='client.49144 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi016-74285-55"}]: dispatch 2024-03-26T04:23:32.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:32 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/535223606' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi016-74285-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:32.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:32 smithi016 ceph-mon[37215]: from='client.49144 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi016-74285-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:32.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:32 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:23:32.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:32 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:23:32.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:32 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:23:32.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:32 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:23:32.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:32 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:23:32.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:32 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:23:32.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:32 smithi016 ceph-mon[29101]: pgmap v389: 292 pgs: 11 creating+peering, 21 unknown, 260 active+clean; 4.6 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T04:23:32.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:32 smithi016 ceph-mon[29101]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:23:32.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:32 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-39", "mode": "writeback"}]': finished 2024-03-26T04:23:32.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:32 smithi016 ceph-mon[29101]: from='client.56615 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi016-74285-54"}]': finished 2024-03-26T04:23:32.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:32 smithi016 ceph-mon[29101]: osdmap e287: 8 total, 8 up, 8 in 2024-03-26T04:23:32.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:32 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/535223606' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi016-74285-55"}]: dispatch 2024-03-26T04:23:32.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:32 smithi016 ceph-mon[29101]: from='client.49144 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi016-74285-55"}]: dispatch 2024-03-26T04:23:32.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:32 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/535223606' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi016-74285-55"}]: dispatch 2024-03-26T04:23:32.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:32 smithi016 ceph-mon[29101]: from='client.49144 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi016-74285-55"}]: dispatch 2024-03-26T04:23:32.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:32 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/535223606' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi016-74285-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:32.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:32 smithi016 ceph-mon[29101]: from='client.49144 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi016-74285-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:32.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:32 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:23:32.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:32 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:23:32.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:32 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:23:32.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:32 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:23:32.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:32 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:23:32.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:32 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:23:32.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:23:32 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:23:32.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:32 smithi183 ceph-mon[33745]: pgmap v389: 292 pgs: 11 creating+peering, 21 unknown, 260 active+clean; 4.6 MiB data, 595 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T04:23:32.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:32 smithi183 ceph-mon[33745]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:23:32.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:32 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-39", "mode": "writeback"}]': finished 2024-03-26T04:23:32.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:32 smithi183 ceph-mon[33745]: from='client.56615 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsCompletePP_smithi016-74285-54"}]': finished 2024-03-26T04:23:32.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:32 smithi183 ceph-mon[33745]: osdmap e287: 8 total, 8 up, 8 in 2024-03-26T04:23:32.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:32 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/535223606' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi016-74285-55"}]: dispatch 2024-03-26T04:23:32.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:32 smithi183 ceph-mon[33745]: from='client.49144 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi016-74285-55"}]: dispatch 2024-03-26T04:23:32.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:32 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/535223606' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi016-74285-55"}]: dispatch 2024-03-26T04:23:32.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:32 smithi183 ceph-mon[33745]: from='client.49144 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi016-74285-55"}]: dispatch 2024-03-26T04:23:32.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:32 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/535223606' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi016-74285-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:32.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:32 smithi183 ceph-mon[33745]: from='client.49144 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi016-74285-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:32.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:32 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:23:32.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:32 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:23:32.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:32 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:23:32.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:32 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:23:32.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:32 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:23:32.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:32 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:23:33.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:33 smithi016 ceph-mon[37215]: from='client.49144 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi016-74285-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:23:33.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:33 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:23:33.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:33 smithi016 ceph-mon[37215]: osdmap e288: 8 total, 8 up, 8 in 2024-03-26T04:23:33.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:33 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/535223606' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi016-74285-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi016-74285-55"}]: dispatch 2024-03-26T04:23:33.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:33 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-39"}]: dispatch 2024-03-26T04:23:33.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:33 smithi016 ceph-mon[37215]: from='client.49144 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi016-74285-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi016-74285-55"}]: dispatch 2024-03-26T04:23:33.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:33 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-39"}]: dispatch 2024-03-26T04:23:33.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:33 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:23:33.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:33 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:23:33.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:33 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:23:33.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:33 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:23:33.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:33 smithi016 ceph-mon[29101]: from='client.49144 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi016-74285-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:23:33.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:33 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:23:33.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:33 smithi016 ceph-mon[29101]: osdmap e288: 8 total, 8 up, 8 in 2024-03-26T04:23:33.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:33 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/535223606' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi016-74285-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi016-74285-55"}]: dispatch 2024-03-26T04:23:33.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:33 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-39"}]: dispatch 2024-03-26T04:23:33.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:33 smithi016 ceph-mon[29101]: from='client.49144 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi016-74285-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi016-74285-55"}]: dispatch 2024-03-26T04:23:33.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:33 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-39"}]: dispatch 2024-03-26T04:23:33.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:33 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:23:33.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:33 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:23:33.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:33 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:23:33.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:33 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:23:33.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:33 smithi183 ceph-mon[33745]: from='client.49144 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-IsSafePP_smithi016-74285-55", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:23:33.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:33 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:23:33.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:33 smithi183 ceph-mon[33745]: osdmap e288: 8 total, 8 up, 8 in 2024-03-26T04:23:33.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:33 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/535223606' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi016-74285-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi016-74285-55"}]: dispatch 2024-03-26T04:23:33.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:33 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-39"}]: dispatch 2024-03-26T04:23:33.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:33 smithi183 ceph-mon[33745]: from='client.49144 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "IsSafePP_smithi016-74285-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi016-74285-55"}]: dispatch 2024-03-26T04:23:33.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:33 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-39"}]: dispatch 2024-03-26T04:23:33.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:33 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:23:33.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:33 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:23:33.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:33 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:23:33.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:33 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:23:34.143 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-03-26T04:23:34.143 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-03-26T04:23:34.143 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-03-26T04:23:34.143 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: seed 74534 2024-03-26T04:23:34.144 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-03-26T04:23:34.144 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (327 ms) 2024-03-26T04:23:34.144 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-03-26T04:23:34.144 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (18957 ms) 2024-03-26T04:23:34.144 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-03-26T04:23:34.144 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (31 ms) 2024-03-26T04:23:34.144 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (19315 ms total) 2024-03-26T04:23:34.144 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: 2024-03-26T04:23:34.144 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-03-26T04:23:34.144 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-03-26T04:23:34.144 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (8057 ms) 2024-03-26T04:23:34.144 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-03-26T04:23:34.144 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (7974 ms) 2024-03-26T04:23:34.144 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-03-26T04:23:34.145 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (10010 ms) 2024-03-26T04:23:34.145 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-03-26T04:23:34.145 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: my_snaps [3] 2024-03-26T04:23:34.145 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: my_snaps [4,3] 2024-03-26T04:23:34.145 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: my_snaps [5,4,3] 2024-03-26T04:23:34.145 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: my_snaps [6,5,4,3] 2024-03-26T04:23:34.145 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: promoting some heads 2024-03-26T04:23:34.145 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: promoting from clones for snap 6 2024-03-26T04:23:34.145 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: promoting from clones for snap 5 2024-03-26T04:23:34.145 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: promoting from clones for snap 4 2024-03-26T04:23:34.145 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: promoting from clones for snap 3 2024-03-26T04:23:34.145 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: waiting for scrubs... 2024-03-26T04:23:34.145 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: done waiting 2024-03-26T04:23:34.145 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (46009 ms) 2024-03-26T04:23:34.145 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-03-26T04:23:34.146 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (10060 ms) 2024-03-26T04:23:34.146 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-03-26T04:23:34.146 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (7950 ms) 2024-03-26T04:23:34.146 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-03-26T04:23:34.146 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (8295 ms) 2024-03-26T04:23:34.146 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-03-26T04:23:34.146 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (7767 ms) 2024-03-26T04:23:34.146 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-03-26T04:23:34.146 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (10032 ms) 2024-03-26T04:23:34.146 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-03-26T04:23:34.146 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (8957 ms) 2024-03-26T04:23:34.146 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-03-26T04:23:34.146 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (10015 ms) 2024-03-26T04:23:34.146 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-03-26T04:23:34.147 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (12201 ms) 2024-03-26T04:23:34.147 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-03-26T04:23:34.147 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (7767 ms) 2024-03-26T04:23:34.147 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-03-26T04:23:34.147 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (8019 ms) 2024-03-26T04:23:34.147 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-03-26T04:23:34.147 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (13035 ms) 2024-03-26T04:23:34.147 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-03-26T04:23:34.147 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (8187 ms) 2024-03-26T04:23:34.147 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-03-26T04:23:34.147 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (7775 ms) 2024-03-26T04:23:34.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:34 smithi016 ceph-mon[37215]: pgmap v392: 292 pgs: 292 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 69 op/s 2024-03-26T04:23:34.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:34 smithi016 ceph-mon[37215]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:23:34.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:34 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-39"}]': finished 2024-03-26T04:23:34.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:34 smithi016 ceph-mon[37215]: osdmap e289: 8 total, 8 up, 8 in 2024-03-26T04:23:34.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:34 smithi016 ceph-mon[37215]: from='client.49144 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi016-74285-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi016-74285-55"}]': finished 2024-03-26T04:23:34.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:34 smithi016 ceph-mon[37215]: osdmap e290: 8 total, 8 up, 8 in 2024-03-26T04:23:34.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:34 smithi016 ceph-mon[29101]: pgmap v392: 292 pgs: 292 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 69 op/s 2024-03-26T04:23:34.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:34 smithi016 ceph-mon[29101]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:23:34.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:34 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-39"}]': finished 2024-03-26T04:23:34.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:34 smithi016 ceph-mon[29101]: osdmap e289: 8 total, 8 up, 8 in 2024-03-26T04:23:34.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:34 smithi016 ceph-mon[29101]: from='client.49144 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi016-74285-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi016-74285-55"}]': finished 2024-03-26T04:23:34.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:34 smithi016 ceph-mon[29101]: osdmap e290: 8 total, 8 up, 8 in 2024-03-26T04:23:34.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:34 smithi183 ceph-mon[33745]: pgmap v392: 292 pgs: 292 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 69 op/s 2024-03-26T04:23:34.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:34 smithi183 ceph-mon[33745]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:23:34.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:34 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-39"}]': finished 2024-03-26T04:23:34.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:34 smithi183 ceph-mon[33745]: osdmap e289: 8 total, 8 up, 8 in 2024-03-26T04:23:34.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:34 smithi183 ceph-mon[33745]: from='client.49144 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "IsSafePP_smithi016-74285-55", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-IsSafePP_smithi016-74285-55"}]': finished 2024-03-26T04:23:34.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:34 smithi183 ceph-mon[33745]: osdmap e290: 8 total, 8 up, 8 in 2024-03-26T04:23:36.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:36 smithi016 ceph-mon[29101]: pgmap v395: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-26T04:23:36.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:36 smithi016 ceph-mon[29101]: osdmap e291: 8 total, 8 up, 8 in 2024-03-26T04:23:36.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:36 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:23:36.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:36 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:23:36.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:36 smithi016 ceph-mon[29101]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:23:36.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:36 smithi016 ceph-mon[37215]: pgmap v395: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-26T04:23:36.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:36 smithi016 ceph-mon[37215]: osdmap e291: 8 total, 8 up, 8 in 2024-03-26T04:23:36.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:36 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:23:36.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:36 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:23:36.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:36 smithi016 ceph-mon[37215]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:23:36.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:36 smithi183 ceph-mon[33745]: pgmap v395: 268 pgs: 8 unknown, 260 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 976 KiB/s wr, 1 op/s 2024-03-26T04:23:36.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:36 smithi183 ceph-mon[33745]: osdmap e291: 8 total, 8 up, 8 in 2024-03-26T04:23:36.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:36 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:23:36.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:36 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-41","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:23:36.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:36 smithi183 ceph-mon[33745]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:23:37.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:37 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:23:37.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:37 smithi016 ceph-mon[29101]: osdmap e292: 8 total, 8 up, 8 in 2024-03-26T04:23:37.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:37 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:23:37.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:37 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/535223606' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi016-74285-55"}]: dispatch 2024-03-26T04:23:37.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:37 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:23:37.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:37 smithi016 ceph-mon[29101]: from='client.49144 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi016-74285-55"}]: dispatch 2024-03-26T04:23:37.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:37 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:23:37.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:37 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:23:37.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:37 smithi016 ceph-mon[37215]: osdmap e292: 8 total, 8 up, 8 in 2024-03-26T04:23:37.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:37 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:23:37.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:37 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/535223606' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi016-74285-55"}]: dispatch 2024-03-26T04:23:37.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:37 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:23:37.415 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:37 smithi016 ceph-mon[37215]: from='client.49144 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi016-74285-55"}]: dispatch 2024-03-26T04:23:37.416 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:37 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:23:37.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:37 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-41","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:23:37.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:37 smithi183 ceph-mon[33745]: osdmap e292: 8 total, 8 up, 8 in 2024-03-26T04:23:37.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:37 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:23:37.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:37 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/535223606' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi016-74285-55"}]: dispatch 2024-03-26T04:23:37.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:37 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-41", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:23:37.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:37 smithi183 ceph-mon[33745]: from='client.49144 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi016-74285-55"}]: dispatch 2024-03-26T04:23:37.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:37 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:23:38.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:38 smithi016 ceph-mon[29101]: pgmap v398: 292 pgs: 26 unknown, 266 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:23:38.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:38 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-41", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:23:38.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:38 smithi016 ceph-mon[29101]: from='client.49144 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi016-74285-55"}]': finished 2024-03-26T04:23:38.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:38 smithi016 ceph-mon[29101]: osdmap e293: 8 total, 8 up, 8 in 2024-03-26T04:23:38.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:38 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:23:38.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:38 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/535223606' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi016-74285-55"}]: dispatch 2024-03-26T04:23:38.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:38 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:23:38.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:38 smithi016 ceph-mon[29101]: from='client.49144 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi016-74285-55"}]: dispatch 2024-03-26T04:23:38.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:38 smithi016 ceph-mon[37215]: pgmap v398: 292 pgs: 26 unknown, 266 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:23:38.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:38 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-41", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:23:38.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:38 smithi016 ceph-mon[37215]: from='client.49144 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi016-74285-55"}]': finished 2024-03-26T04:23:38.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:38 smithi016 ceph-mon[37215]: osdmap e293: 8 total, 8 up, 8 in 2024-03-26T04:23:38.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:38 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:23:38.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:38 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/535223606' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi016-74285-55"}]: dispatch 2024-03-26T04:23:38.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:38 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:23:38.414 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:38 smithi016 ceph-mon[37215]: from='client.49144 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi016-74285-55"}]: dispatch 2024-03-26T04:23:38.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:38 smithi183 ceph-mon[33745]: pgmap v398: 292 pgs: 26 unknown, 266 active+clean; 8.4 MiB data, 611 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:23:38.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:38 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-41", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:23:38.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:38 smithi183 ceph-mon[33745]: from='client.49144 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-IsSafePP_smithi016-74285-55"}]': finished 2024-03-26T04:23:38.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:38 smithi183 ceph-mon[33745]: osdmap e293: 8 total, 8 up, 8 in 2024-03-26T04:23:38.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:38 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:23:38.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:38 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/535223606' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi016-74285-55"}]: dispatch 2024-03-26T04:23:38.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:38 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-41","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:23:38.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:38 smithi183 ceph-mon[33745]: from='client.49144 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi016-74285-55"}]: dispatch 2024-03-26T04:23:39.161 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:23:38 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:23:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:23:39.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:39 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-41","var": "hit_set_count","val": "2"}]': finished 2024-03-26T04:23:39.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:39 smithi016 ceph-mon[37215]: from='client.49144 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi016-74285-55"}]': finished 2024-03-26T04:23:39.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:39 smithi016 ceph-mon[37215]: osdmap e294: 8 total, 8 up, 8 in 2024-03-26T04:23:39.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:39 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:23:39.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:39 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:23:39.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:39 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3518970319' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi016-74285-56"}]: dispatch 2024-03-26T04:23:39.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:39 smithi016 ceph-mon[37215]: from='client.56627 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi016-74285-56"}]: dispatch 2024-03-26T04:23:39.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:39 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3518970319' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi016-74285-56"}]: dispatch 2024-03-26T04:23:39.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:39 smithi016 ceph-mon[37215]: from='client.56627 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi016-74285-56"}]: dispatch 2024-03-26T04:23:39.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:39 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3518970319' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi016-74285-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:39.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:39 smithi016 ceph-mon[37215]: from='client.56627 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi016-74285-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:39.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:39 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-41","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:23:39.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:39 smithi016 ceph-mon[37215]: from='client.56627 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi016-74285-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:23:39.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:39 smithi016 ceph-mon[37215]: osdmap e295: 8 total, 8 up, 8 in 2024-03-26T04:23:39.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:39 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-41","var": "hit_set_count","val": "2"}]': finished 2024-03-26T04:23:39.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:39 smithi016 ceph-mon[29101]: from='client.49144 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi016-74285-55"}]': finished 2024-03-26T04:23:39.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:39 smithi016 ceph-mon[29101]: osdmap e294: 8 total, 8 up, 8 in 2024-03-26T04:23:39.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:39 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:23:39.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:39 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:23:39.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:39 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3518970319' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi016-74285-56"}]: dispatch 2024-03-26T04:23:39.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:39 smithi016 ceph-mon[29101]: from='client.56627 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi016-74285-56"}]: dispatch 2024-03-26T04:23:39.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:39 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3518970319' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi016-74285-56"}]: dispatch 2024-03-26T04:23:39.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:39 smithi016 ceph-mon[29101]: from='client.56627 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi016-74285-56"}]: dispatch 2024-03-26T04:23:39.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:39 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3518970319' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi016-74285-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:39.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:39 smithi016 ceph-mon[29101]: from='client.56627 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi016-74285-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:39.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:39 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-41","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:23:39.414 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:39 smithi016 ceph-mon[29101]: from='client.56627 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi016-74285-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:23:39.415 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:39 smithi016 ceph-mon[29101]: osdmap e295: 8 total, 8 up, 8 in 2024-03-26T04:23:39.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:39 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-41","var": "hit_set_count","val": "2"}]': finished 2024-03-26T04:23:39.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:39 smithi183 ceph-mon[33745]: from='client.49144 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"IsSafePP_smithi016-74285-55"}]': finished 2024-03-26T04:23:39.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:39 smithi183 ceph-mon[33745]: osdmap e294: 8 total, 8 up, 8 in 2024-03-26T04:23:39.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:39 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:23:39.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:39 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-41","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:23:39.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:39 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3518970319' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi016-74285-56"}]: dispatch 2024-03-26T04:23:39.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:39 smithi183 ceph-mon[33745]: from='client.56627 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi016-74285-56"}]: dispatch 2024-03-26T04:23:39.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:39 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3518970319' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi016-74285-56"}]: dispatch 2024-03-26T04:23:39.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:39 smithi183 ceph-mon[33745]: from='client.56627 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi016-74285-56"}]: dispatch 2024-03-26T04:23:39.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:39 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3518970319' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi016-74285-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:39.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:39 smithi183 ceph-mon[33745]: from='client.56627 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi016-74285-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:39.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:39 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-41","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:23:39.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:39 smithi183 ceph-mon[33745]: from='client.56627 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReturnValuePP_smithi016-74285-56", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:23:39.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:39 smithi183 ceph-mon[33745]: osdmap e295: 8 total, 8 up, 8 in 2024-03-26T04:23:40.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:40 smithi183 ceph-mon[33745]: pgmap v401: 292 pgs: 292 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:23:40.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:40 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3518970319' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi016-74285-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi016-74285-56"}]: dispatch 2024-03-26T04:23:40.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:40 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T04:23:40.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:40 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T04:23:40.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:40 smithi183 ceph-mon[33745]: from='client.56627 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi016-74285-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi016-74285-56"}]: dispatch 2024-03-26T04:23:40.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:40 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T04:23:40.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:40 smithi016 ceph-mon[37215]: pgmap v401: 292 pgs: 292 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:23:40.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:40 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3518970319' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi016-74285-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi016-74285-56"}]: dispatch 2024-03-26T04:23:40.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:40 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T04:23:40.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:40 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T04:23:40.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:40 smithi016 ceph-mon[37215]: from='client.56627 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi016-74285-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi016-74285-56"}]: dispatch 2024-03-26T04:23:40.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:40 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T04:23:40.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:40 smithi016 ceph-mon[29101]: pgmap v401: 292 pgs: 292 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:23:40.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:40 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3518970319' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi016-74285-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi016-74285-56"}]: dispatch 2024-03-26T04:23:40.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:40 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T04:23:40.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:40 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-41","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T04:23:40.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:40 smithi016 ceph-mon[29101]: from='client.56627 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi016-74285-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi016-74285-56"}]: dispatch 2024-03-26T04:23:40.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:40 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-41","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T04:23:41.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:41 smithi183 ceph-mon[33745]: osdmap e296: 8 total, 8 up, 8 in 2024-03-26T04:23:41.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:41 smithi183 ceph-mon[33745]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:23:41.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:41 smithi183 ceph-mon[33745]: from='client.56627 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi016-74285-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi016-74285-56"}]': finished 2024-03-26T04:23:41.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:41 smithi183 ceph-mon[33745]: osdmap e297: 8 total, 8 up, 8 in 2024-03-26T04:23:41.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:41 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:23:41.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:41 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:23:41.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:41 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-41"}]: dispatch 2024-03-26T04:23:41.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:41 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-41"}]: dispatch 2024-03-26T04:23:41.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:41 smithi016 ceph-mon[37215]: osdmap e296: 8 total, 8 up, 8 in 2024-03-26T04:23:41.952 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:41 smithi016 ceph-mon[37215]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:23:41.952 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:41 smithi016 ceph-mon[37215]: from='client.56627 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi016-74285-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi016-74285-56"}]': finished 2024-03-26T04:23:41.952 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:41 smithi016 ceph-mon[37215]: osdmap e297: 8 total, 8 up, 8 in 2024-03-26T04:23:41.952 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:41 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:23:41.952 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:41 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:23:41.952 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:41 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-41"}]: dispatch 2024-03-26T04:23:41.952 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:41 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-41"}]: dispatch 2024-03-26T04:23:41.952 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:41 smithi016 ceph-mon[29101]: osdmap e296: 8 total, 8 up, 8 in 2024-03-26T04:23:41.952 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:41 smithi016 ceph-mon[29101]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:23:41.953 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:41 smithi016 ceph-mon[29101]: from='client.56627 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ReturnValuePP_smithi016-74285-56", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReturnValuePP_smithi016-74285-56"}]': finished 2024-03-26T04:23:41.953 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:41 smithi016 ceph-mon[29101]: osdmap e297: 8 total, 8 up, 8 in 2024-03-26T04:23:41.953 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:41 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:23:41.953 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:41 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:23:41.953 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:41 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-41"}]: dispatch 2024-03-26T04:23:41.953 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:41 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-41"}]: dispatch 2024-03-26T04:23:42.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:23:42 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:23:42.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:42 smithi183 ceph-mon[33745]: pgmap v404: 292 pgs: 292 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:23:42.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:42 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-41"}]': finished 2024-03-26T04:23:42.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:42 smithi183 ceph-mon[33745]: osdmap e298: 8 total, 8 up, 8 in 2024-03-26T04:23:42.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:42 smithi016 ceph-mon[37215]: pgmap v404: 292 pgs: 292 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:23:42.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:42 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-41"}]': finished 2024-03-26T04:23:42.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:42 smithi016 ceph-mon[37215]: osdmap e298: 8 total, 8 up, 8 in 2024-03-26T04:23:42.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:42 smithi016 ceph-mon[29101]: pgmap v404: 292 pgs: 292 active+clean; 8.4 MiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:23:42.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:42 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-41"}]': finished 2024-03-26T04:23:42.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:42 smithi016 ceph-mon[29101]: osdmap e298: 8 total, 8 up, 8 in 2024-03-26T04:23:43.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:43 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:23:43.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:43 smithi183 ceph-mon[33745]: osdmap e299: 8 total, 8 up, 8 in 2024-03-26T04:23:43.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:43 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3518970319' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi016-74285-56"}]: dispatch 2024-03-26T04:23:43.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:43 smithi183 ceph-mon[33745]: from='client.56627 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi016-74285-56"}]: dispatch 2024-03-26T04:23:43.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:43 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:23:43.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:43 smithi016 ceph-mon[29101]: osdmap e299: 8 total, 8 up, 8 in 2024-03-26T04:23:43.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:43 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3518970319' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi016-74285-56"}]: dispatch 2024-03-26T04:23:43.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:43 smithi016 ceph-mon[29101]: from='client.56627 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi016-74285-56"}]: dispatch 2024-03-26T04:23:43.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:43 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:23:43.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:43 smithi016 ceph-mon[37215]: osdmap e299: 8 total, 8 up, 8 in 2024-03-26T04:23:43.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:43 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3518970319' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi016-74285-56"}]: dispatch 2024-03-26T04:23:43.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:43 smithi016 ceph-mon[37215]: from='client.56627 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi016-74285-56"}]: dispatch 2024-03-26T04:23:44.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:44 smithi183 ceph-mon[33745]: pgmap v407: 300 pgs: 3 creating+peering, 297 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:23:44.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:44 smithi183 ceph-mon[33745]: from='client.56627 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi016-74285-56"}]': finished 2024-03-26T04:23:44.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:44 smithi183 ceph-mon[33745]: osdmap e300: 8 total, 8 up, 8 in 2024-03-26T04:23:44.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:44 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3518970319' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi016-74285-56"}]: dispatch 2024-03-26T04:23:44.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:44 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:23:44.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:44 smithi183 ceph-mon[33745]: from='client.56627 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi016-74285-56"}]: dispatch 2024-03-26T04:23:44.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:44 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:23:44.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:44 smithi016 ceph-mon[29101]: pgmap v407: 300 pgs: 3 creating+peering, 297 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:23:44.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:44 smithi016 ceph-mon[29101]: from='client.56627 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi016-74285-56"}]': finished 2024-03-26T04:23:44.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:44 smithi016 ceph-mon[29101]: osdmap e300: 8 total, 8 up, 8 in 2024-03-26T04:23:44.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:44 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3518970319' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi016-74285-56"}]: dispatch 2024-03-26T04:23:44.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:44 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:23:44.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:44 smithi016 ceph-mon[29101]: from='client.56627 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi016-74285-56"}]: dispatch 2024-03-26T04:23:44.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:44 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:23:44.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:44 smithi016 ceph-mon[37215]: pgmap v407: 300 pgs: 3 creating+peering, 297 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:23:44.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:44 smithi016 ceph-mon[37215]: from='client.56627 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ReturnValuePP_smithi016-74285-56"}]': finished 2024-03-26T04:23:44.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:44 smithi016 ceph-mon[37215]: osdmap e300: 8 total, 8 up, 8 in 2024-03-26T04:23:44.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:44 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3518970319' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi016-74285-56"}]: dispatch 2024-03-26T04:23:44.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:44 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:23:44.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:44 smithi016 ceph-mon[37215]: from='client.56627 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi016-74285-56"}]: dispatch 2024-03-26T04:23:44.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:44 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-43","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:23:46.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:46 smithi183 ceph-mon[33745]: pgmap v410: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:23:46.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:46 smithi183 ceph-mon[33745]: from='client.56627 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi016-74285-56"}]': finished 2024-03-26T04:23:46.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:46 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:23:46.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:46 smithi183 ceph-mon[33745]: osdmap e301: 8 total, 8 up, 8 in 2024-03-26T04:23:46.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:46 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi016-74534-6","var": "pg_num","format": "json"}]: dispatch 2024-03-26T04:23:46.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:46 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:23:46.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:46 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:23:46.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:46 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1963457478' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi016-74285-57"}]: dispatch 2024-03-26T04:23:46.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:46 smithi183 ceph-mon[33745]: from='client.56633 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi016-74285-57"}]: dispatch 2024-03-26T04:23:46.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:46 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1963457478' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi016-74285-57"}]: dispatch 2024-03-26T04:23:46.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:46 smithi183 ceph-mon[33745]: from='client.56633 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi016-74285-57"}]: dispatch 2024-03-26T04:23:46.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:46 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1963457478' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi016-74285-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:46.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:46 smithi183 ceph-mon[33745]: from='client.56633 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi016-74285-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:46.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:46 smithi183 ceph-mon[33745]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:23:46.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:46 smithi016 ceph-mon[37215]: pgmap v410: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:23:46.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:46 smithi016 ceph-mon[37215]: from='client.56627 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi016-74285-56"}]': finished 2024-03-26T04:23:46.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:46 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:23:46.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:46 smithi016 ceph-mon[37215]: osdmap e301: 8 total, 8 up, 8 in 2024-03-26T04:23:46.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:46 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi016-74534-6","var": "pg_num","format": "json"}]: dispatch 2024-03-26T04:23:46.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:46 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:23:46.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:46 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:23:46.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:46 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1963457478' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi016-74285-57"}]: dispatch 2024-03-26T04:23:46.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:46 smithi016 ceph-mon[37215]: from='client.56633 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi016-74285-57"}]: dispatch 2024-03-26T04:23:46.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:46 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1963457478' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi016-74285-57"}]: dispatch 2024-03-26T04:23:46.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:46 smithi016 ceph-mon[37215]: from='client.56633 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi016-74285-57"}]: dispatch 2024-03-26T04:23:46.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:46 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1963457478' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi016-74285-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:46.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:46 smithi016 ceph-mon[37215]: from='client.56633 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi016-74285-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:46.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:46 smithi016 ceph-mon[37215]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:23:46.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:46 smithi016 ceph-mon[29101]: pgmap v410: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:23:46.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:46 smithi016 ceph-mon[29101]: from='client.56627 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ReturnValuePP_smithi016-74285-56"}]': finished 2024-03-26T04:23:46.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:46 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-43","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:23:46.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:46 smithi016 ceph-mon[29101]: osdmap e301: 8 total, 8 up, 8 in 2024-03-26T04:23:46.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:46 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi016-74534-6","var": "pg_num","format": "json"}]: dispatch 2024-03-26T04:23:46.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:46 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:23:46.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:46 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-43", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:23:46.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:46 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1963457478' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi016-74285-57"}]: dispatch 2024-03-26T04:23:46.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:46 smithi016 ceph-mon[29101]: from='client.56633 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi016-74285-57"}]: dispatch 2024-03-26T04:23:46.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:46 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1963457478' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi016-74285-57"}]: dispatch 2024-03-26T04:23:46.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:46 smithi016 ceph-mon[29101]: from='client.56633 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi016-74285-57"}]: dispatch 2024-03-26T04:23:46.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:46 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1963457478' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi016-74285-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:46.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:46 smithi016 ceph-mon[29101]: from='client.56633 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi016-74285-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:46.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:46 smithi016 ceph-mon[29101]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:23:47.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:47 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-43", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:23:47.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:47 smithi183 ceph-mon[33745]: from='client.56633 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi016-74285-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:23:47.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:47 smithi183 ceph-mon[33745]: osdmap e302: 8 total, 8 up, 8 in 2024-03-26T04:23:47.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:47 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-26T04:23:47.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:47 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1963457478' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi016-74285-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi016-74285-57"}]: dispatch 2024-03-26T04:23:47.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:47 smithi183 ceph-mon[33745]: from='client.56633 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi016-74285-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi016-74285-57"}]: dispatch 2024-03-26T04:23:47.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:47 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-26T04:23:47.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:47 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-43", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:23:47.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:47 smithi016 ceph-mon[29101]: from='client.56633 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi016-74285-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:23:47.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:47 smithi016 ceph-mon[29101]: osdmap e302: 8 total, 8 up, 8 in 2024-03-26T04:23:47.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:47 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-26T04:23:47.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:47 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1963457478' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi016-74285-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi016-74285-57"}]: dispatch 2024-03-26T04:23:47.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:47 smithi016 ceph-mon[29101]: from='client.56633 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi016-74285-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi016-74285-57"}]: dispatch 2024-03-26T04:23:47.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:47 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-26T04:23:47.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:47 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-43", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:23:47.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:47 smithi016 ceph-mon[37215]: from='client.56633 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushPP_smithi016-74285-57", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:23:47.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:47 smithi016 ceph-mon[37215]: osdmap e302: 8 total, 8 up, 8 in 2024-03-26T04:23:47.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:47 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-26T04:23:47.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:47 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1963457478' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi016-74285-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi016-74285-57"}]: dispatch 2024-03-26T04:23:47.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:47 smithi016 ceph-mon[37215]: from='client.56633 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushPP_smithi016-74285-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi016-74285-57"}]: dispatch 2024-03-26T04:23:47.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:47 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-43","var": "hit_set_count","val": "8"}]: dispatch 2024-03-26T04:23:48.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:48 smithi183 ceph-mon[33745]: pgmap v413: 292 pgs: 19 unknown, 273 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:23:48.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:48 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-43","var": "hit_set_count","val": "8"}]': finished 2024-03-26T04:23:48.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:48 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:23:48.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:48 smithi183 ceph-mon[33745]: osdmap e303: 8 total, 8 up, 8 in 2024-03-26T04:23:48.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:48 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:23:48.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:48 smithi016 ceph-mon[29101]: pgmap v413: 292 pgs: 19 unknown, 273 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:23:48.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:48 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-43","var": "hit_set_count","val": "8"}]': finished 2024-03-26T04:23:48.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:48 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:23:48.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:48 smithi016 ceph-mon[29101]: osdmap e303: 8 total, 8 up, 8 in 2024-03-26T04:23:48.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:48 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:23:48.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:48 smithi016 ceph-mon[37215]: pgmap v413: 292 pgs: 19 unknown, 273 active+clean; 8.4 MiB data, 616 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:23:48.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:48 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-43","var": "hit_set_count","val": "8"}]': finished 2024-03-26T04:23:48.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:48 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:23:48.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:48 smithi016 ceph-mon[37215]: osdmap e303: 8 total, 8 up, 8 in 2024-03-26T04:23:48.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:48 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-43","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:23:49.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:23:48 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:23:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:23:49.430 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-03-26T04:23:49.430 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: hmm, no HitSet yet 2024-03-26T04:23:49.430 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: ok, hit_set contains 260:602f83fe:::foo:head 2024-03-26T04:23:49.430 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (9022 ms) 2024-03-26T04:23:49.430 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-03-26T04:23:49.430 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: pg_num = 32 2024-03-26T04:23:49.430 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: pg 0 ls 1711427030,0 2024-03-26T04:23:49.430 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: pg 1 ls 1711427030,0 2024-03-26T04:23:49.430 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: pg 2 ls 1711427030,0 2024-03-26T04:23:49.430 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: pg 3 ls 1711427030,0 2024-03-26T04:23:49.431 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: pg 4 ls 1711427030,0 2024-03-26T04:23:49.431 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: pg 5 ls 1711427030,0 2024-03-26T04:23:49.431 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: pg 6 ls 1711427030,0 2024-03-26T04:23:49.431 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: pg 7 ls 1711427030,0 2024-03-26T04:23:49.431 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: pg 8 ls 1711427030,0 2024-03-26T04:23:49.431 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: pg 9 ls 1711427030,0 2024-03-26T04:23:49.431 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: pg 10 ls 1711427030,0 2024-03-26T04:23:49.431 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: pg 11 ls 1711427030,0 2024-03-26T04:23:49.431 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: pg 12 ls 1711427030,0 2024-03-26T04:23:49.431 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: pg 13 ls 1711427030,0 2024-03-26T04:23:49.431 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: pg 14 ls 1711427030,0 2024-03-26T04:23:49.431 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: pg 15 ls 1711427030,0 2024-03-26T04:23:49.431 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: pg 16 ls 1711427030,0 2024-03-26T04:23:49.431 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: pg 17 ls 1711427030,0 2024-03-26T04:23:49.431 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: pg 18 ls 1711427030,0 2024-03-26T04:23:49.432 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: pg 19 ls 1711427030,0 2024-03-26T04:23:49.432 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: pg 20 ls 1711427030,0 2024-03-26T04:23:49.432 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: pg 21 ls 1711427030,0 2024-03-26T04:23:49.432 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: pg 22 ls 1711427030,0 2024-03-26T04:23:49.432 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: pg 23 ls 1711427030,0 2024-03-26T04:23:49.432 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: pg 24 ls 1711427030,0 2024-03-26T04:23:49.432 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: pg 25 ls 1711427030,0 2024-03-26T04:23:49.432 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: pg 26 ls 1711427030,0 2024-03-26T04:23:49.432 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: pg 27 ls 1711427030,0 2024-03-26T04:23:49.432 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: pg 28 ls 1711427030,0 2024-03-26T04:23:49.432 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: pg 29 ls 1711427030,0 2024-03-26T04:23:49.432 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: pg 30 ls 1711427030,0 2024-03-26T04:23:49.432 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: pg 31 ls 1711427030,0 2024-03-26T04:23:49.432 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: pg_num = 32 2024-03-26T04:23:49.432 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:6cac518f:::0:head 2024-03-26T04:23:49.433 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:02547ec2:::1:head 2024-03-26T04:23:49.433 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:f905c69b:::2:head 2024-03-26T04:23:49.433 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:cfc208b3:::3:head 2024-03-26T04:23:49.433 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:d83876eb:::4:head 2024-03-26T04:23:49.433 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:b29083e3:::5:head 2024-03-26T04:23:49.433 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:c4fdafeb:::6:head 2024-03-26T04:23:49.433 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:5c6b0b28:::7:head 2024-03-26T04:23:49.433 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:bd63b0f1:::8:head 2024-03-26T04:23:49.433 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:e960b815:::9:head 2024-03-26T04:23:49.433 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:52ea6a34:::10:head 2024-03-26T04:23:49.433 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:89d3ae78:::11:head 2024-03-26T04:23:49.433 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:de5d7c5f:::12:head 2024-03-26T04:23:49.433 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:566253c9:::13:head 2024-03-26T04:23:49.433 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:62a1935d:::14:head 2024-03-26T04:23:49.434 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:863748b0:::15:head 2024-03-26T04:23:49.434 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:3958e169:::16:head 2024-03-26T04:23:49.434 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:4d4dabf9:::17:head 2024-03-26T04:23:49.434 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:8391935d:::18:head 2024-03-26T04:23:49.434 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:28883081:::19:head 2024-03-26T04:23:49.434 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:69259c59:::20:head 2024-03-26T04:23:49.434 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:4bdb80b7:::21:head 2024-03-26T04:23:49.434 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:a11c5d71:::22:head 2024-03-26T04:23:49.434 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:271af37b:::23:head 2024-03-26T04:23:49.434 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:95b121be:::24:head 2024-03-26T04:23:49.434 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:58d1031b:::25:head 2024-03-26T04:23:49.434 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:0a050783:::26:head 2024-03-26T04:23:49.434 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:c709704c:::27:head 2024-03-26T04:23:49.434 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:cbe56eaf:::28:head 2024-03-26T04:23:49.435 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:86b4b162:::29:head 2024-03-26T04:23:49.435 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:70d89383:::30:head 2024-03-26T04:23:49.435 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:dd450c7c:::31:head 2024-03-26T04:23:49.435 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:6d5729b1:::32:head 2024-03-26T04:23:49.435 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:c388f3fb:::33:head 2024-03-26T04:23:49.435 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:56cfea31:::34:head 2024-03-26T04:23:49.435 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:9dbc1bf7:::35:head 2024-03-26T04:23:49.435 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:40b74ccd:::36:head 2024-03-26T04:23:49.435 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:4d5aaf42:::37:head 2024-03-26T04:23:49.435 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:920f362c:::38:head 2024-03-26T04:23:49.435 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:6cc53222:::39:head 2024-03-26T04:23:49.435 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:9cad833f:::40:head 2024-03-26T04:23:49.435 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:1ea84d41:::41:head 2024-03-26T04:23:49.435 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:c4480ef6:::42:head 2024-03-26T04:23:49.435 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:a694361e:::43:head 2024-03-26T04:23:49.436 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:d1bd33e9:::44:head 2024-03-26T04:23:49.436 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:ddc2cd5d:::45:head 2024-03-26T04:23:49.436 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:2b782207:::46:head 2024-03-26T04:23:49.436 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:7b187fca:::47:head 2024-03-26T04:23:49.436 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:90ecdf6f:::48:head 2024-03-26T04:23:49.436 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:a5ed95fe:::49:head 2024-03-26T04:23:49.436 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:ea0eaa55:::50:head 2024-03-26T04:23:49.436 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:f33ef17b:::51:head 2024-03-26T04:23:49.436 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:a0d1b2f6:::52:head 2024-03-26T04:23:49.436 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:60c5229e:::53:head 2024-03-26T04:23:49.436 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:edcbc575:::54:head 2024-03-26T04:23:49.436 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:102cf253:::55:head 2024-03-26T04:23:49.436 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:efb7fb0b:::56:head 2024-03-26T04:23:49.436 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:50d0a326:::57:head 2024-03-26T04:23:49.437 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:d4dc5daf:::58:head 2024-03-26T04:23:49.437 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:3a130462:::59:head 2024-03-26T04:23:49.437 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:ec87ed71:::60:head 2024-03-26T04:23:49.437 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:d5bc9454:::61:head 2024-03-26T04:23:49.437 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:3ddfe313:::62:head 2024-03-26T04:23:49.437 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:7c2816b9:::63:head 2024-03-26T04:23:49.437 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:47e00e4d:::64:head 2024-03-26T04:23:49.437 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:c6410c18:::65:head 2024-03-26T04:23:49.437 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:b48ed237:::66:head 2024-03-26T04:23:49.437 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:cd63ad31:::67:head 2024-03-26T04:23:49.437 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:b179e92b:::68:head 2024-03-26T04:23:49.437 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:0d9f741a:::69:head 2024-03-26T04:23:49.437 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:6d3352ae:::70:head 2024-03-26T04:23:49.437 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:c6d5c19e:::71:head 2024-03-26T04:23:49.437 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:bc4729c3:::72:head 2024-03-26T04:23:49.438 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:77e930b9:::73:head 2024-03-26T04:23:49.438 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:0abeecfd:::74:head 2024-03-26T04:23:49.438 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:b7c37e15:::75:head 2024-03-26T04:23:49.438 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:b6378398:::76:head 2024-03-26T04:23:49.438 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:02bd68de:::77:head 2024-03-26T04:23:49.438 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:cc795d2d:::78:head 2024-03-26T04:23:49.438 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:630d4fea:::79:head 2024-03-26T04:23:49.438 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:e0d29ef5:::80:head 2024-03-26T04:23:49.438 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:fd6f13d2:::81:head 2024-03-26T04:23:49.438 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:606461d5:::82:head 2024-03-26T04:23:49.438 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:eadbdc43:::83:head 2024-03-26T04:23:49.438 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:8761d0bb:::84:head 2024-03-26T04:23:49.438 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:9ef0186f:::85:head 2024-03-26T04:23:49.438 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:e0d41294:::86:head 2024-03-26T04:23:49.439 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:961de695:::87:head 2024-03-26T04:23:49.439 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:1423148f:::88:head 2024-03-26T04:23:49.439 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:633a8fa2:::89:head 2024-03-26T04:23:49.439 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:a8653809:::90:head 2024-03-26T04:23:49.439 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:3dac8b33:::91:head 2024-03-26T04:23:49.439 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:35aad435:::92:head 2024-03-26T04:23:49.439 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:f6dcc343:::93:head 2024-03-26T04:23:49.439 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:dbbdad87:::94:head 2024-03-26T04:23:49.439 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:1cb48ce0:::95:head 2024-03-26T04:23:49.439 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:03cd461c:::96:head 2024-03-26T04:23:49.439 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:17a4ea99:::97:head 2024-03-26T04:23:49.439 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:9993c9a7:::98:head 2024-03-26T04:23:49.439 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:6394211c:::99:head 2024-03-26T04:23:49.439 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:94c7ae57:::100:head 2024-03-26T04:23:49.439 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:6fdee5bb:::101:head 2024-03-26T04:23:49.440 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:9a477fd1:::102:head 2024-03-26T04:23:49.440 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:eb850916:::103:head 2024-03-26T04:23:49.440 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:affc56b9:::104:head 2024-03-26T04:23:49.440 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:b42dc814:::105:head 2024-03-26T04:23:49.440 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:f319f8f0:::106:head 2024-03-26T04:23:49.440 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:9a40b9de:::107:head 2024-03-26T04:23:49.440 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:8b524f28:::108:head 2024-03-26T04:23:49.440 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:e3de589f:::109:head 2024-03-26T04:23:49.440 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:90f90a5b:::110:head 2024-03-26T04:23:49.440 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:a7b4f1d7:::111:head 2024-03-26T04:23:49.440 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:af51766e:::112:head 2024-03-26T04:23:49.440 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:b6f90bd1:::113:head 2024-03-26T04:23:49.440 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:e0261208:::114:head 2024-03-26T04:23:49.440 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:c9569ef7:::115:head 2024-03-26T04:23:49.440 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:61bebe50:::116:head 2024-03-26T04:23:49.441 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:fe93412b:::117:head 2024-03-26T04:23:49.441 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:d3d38bee:::118:head 2024-03-26T04:23:49.441 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:3100ba0c:::119:head 2024-03-26T04:23:49.441 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:d0560ada:::120:head 2024-03-26T04:23:49.441 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:f0ea8b35:::121:head 2024-03-26T04:23:49.441 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:766f231a:::122:head 2024-03-26T04:23:49.441 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:a07a2582:::123:head 2024-03-26T04:23:49.441 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:bd7c6b3a:::124:head 2024-03-26T04:23:49.441 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:fb2ddaff:::125:head 2024-03-26T04:23:49.441 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:4408e1fe:::126:head 2024-03-26T04:23:49.441 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:ee1df7a7:::127:head 2024-03-26T04:23:49.441 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:c3002909:::128:head 2024-03-26T04:23:49.441 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:4f48ffa9:::129:head 2024-03-26T04:23:49.441 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:edf38733:::130:head 2024-03-26T04:23:49.442 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:c08425c0:::131:head 2024-03-26T04:23:49.442 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:5f902d98:::132:head 2024-03-26T04:23:49.442 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:41ea2c93:::133:head 2024-03-26T04:23:49.442 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:813cee13:::134:head 2024-03-26T04:23:49.442 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:0131818d:::135:head 2024-03-26T04:23:49.442 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:26ba5a85:::136:head 2024-03-26T04:23:49.442 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:381b8a5a:::137:head 2024-03-26T04:23:49.442 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:28797e47:::138:head 2024-03-26T04:23:49.442 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:bfca7f22:::139:head 2024-03-26T04:23:49.442 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:36807075:::140:head 2024-03-26T04:23:49.442 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:80b03975:::141:head 2024-03-26T04:23:49.442 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:5c15709b:::142:head 2024-03-26T04:23:49.442 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:f39ea15e:::143:head 2024-03-26T04:23:49.443 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:ea992956:::144:head 2024-03-26T04:23:49.443 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:48887b1c:::145:head 2024-03-26T04:23:49.443 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:9f24a9dd:::146:head 2024-03-26T04:23:49.443 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:987f100b:::147:head 2024-03-26T04:23:49.443 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:d2dd3581:::148:head 2024-03-26T04:23:49.443 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:7fed1808:::149:head 2024-03-26T04:23:49.443 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:c80b70e9:::150:head 2024-03-26T04:23:49.443 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:85ed90f9:::151:head 2024-03-26T04:23:49.443 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:36428b24:::152:head 2024-03-26T04:23:49.443 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:d044c34a:::153:head 2024-03-26T04:23:49.443 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:7c18bf58:::154:head 2024-03-26T04:23:49.443 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:d1c21232:::155:head 2024-03-26T04:23:49.443 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:a7a3c575:::156:head 2024-03-26T04:23:49.443 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:87da0633:::157:head 2024-03-26T04:23:49.444 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:d5ac3822:::158:head 2024-03-26T04:23:49.444 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:3f20522d:::159:head 2024-03-26T04:23:49.444 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:6ca26563:::160:head 2024-03-26T04:23:49.444 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:532ce135:::161:head 2024-03-26T04:23:49.444 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:c78863e6:::162:head 2024-03-26T04:23:49.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:49 smithi183 ceph-mon[33745]: from='client.56633 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi016-74285-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi016-74285-57"}]': finished 2024-03-26T04:23:49.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:49 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-43","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:23:49.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:49 smithi183 ceph-mon[33745]: osdmap e304: 8 total, 8 up, 8 in 2024-03-26T04:23:49.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:49 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-26T04:23:49.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:49 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-26T04:23:49.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:49 smithi016 ceph-mon[29101]: from='client.56633 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi016-74285-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi016-74285-57"}]': finished 2024-03-26T04:23:49.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:49 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-43","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:23:49.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:49 smithi016 ceph-mon[29101]: osdmap e304: 8 total, 8 up, 8 in 2024-03-26T04:23:49.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:49 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-26T04:23:49.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:49 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-26T04:23:49.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:49 smithi016 ceph-mon[37215]: from='client.56633 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushPP_smithi016-74285-57", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushPP_smithi016-74285-57"}]': finished 2024-03-26T04:23:49.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:49 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-43","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:23:49.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:49 smithi016 ceph-mon[37215]: osdmap e304: 8 total, 8 up, 8 in 2024-03-26T04:23:49.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:49 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-26T04:23:49.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:49 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-43","var": "hit_set_type","val": "explicit_hash"}]: dispatch 2024-03-26T04:23:50.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:50 smithi183 ceph-mon[33745]: pgmap v416: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:23:50.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:50 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-03-26T04:23:50.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:50 smithi183 ceph-mon[33745]: osdmap e305: 8 total, 8 up, 8 in 2024-03-26T04:23:50.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:50 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi016-74534-43","var": "pg_num","format": "json"}]: dispatch 2024-03-26T04:23:50.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:50 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:23:50.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:50 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:23:50.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:50 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-43"}]: dispatch 2024-03-26T04:23:50.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:50 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-43"}]: dispatch 2024-03-26T04:23:50.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:50 smithi016 ceph-mon[29101]: pgmap v416: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:23:50.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:50 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-03-26T04:23:50.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:50 smithi016 ceph-mon[29101]: osdmap e305: 8 total, 8 up, 8 in 2024-03-26T04:23:50.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:50 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi016-74534-43","var": "pg_num","format": "json"}]: dispatch 2024-03-26T04:23:50.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:50 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:23:50.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:50 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:23:50.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:50 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-43"}]: dispatch 2024-03-26T04:23:50.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:50 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-43"}]: dispatch 2024-03-26T04:23:50.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:50 smithi016 ceph-mon[37215]: pgmap v416: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:23:50.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:50 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-43","var": "hit_set_type","val": "explicit_hash"}]': finished 2024-03-26T04:23:50.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:50 smithi016 ceph-mon[37215]: osdmap e305: 8 total, 8 up, 8 in 2024-03-26T04:23:50.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:50 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool get","pool":"test-rados-api-smithi016-74534-43","var": "pg_num","format": "json"}]: dispatch 2024-03-26T04:23:50.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:50 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:23:50.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:50 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:23:50.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:50 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-43"}]: dispatch 2024-03-26T04:23:50.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:50 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-43"}]: dispatch 2024-03-26T04:23:51.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:51 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-43"}]': finished 2024-03-26T04:23:51.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:51 smithi183 ceph-mon[33745]: osdmap e306: 8 total, 8 up, 8 in 2024-03-26T04:23:51.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:51 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1963457478' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi016-74285-57"}]: dispatch 2024-03-26T04:23:51.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:51 smithi183 ceph-mon[33745]: from='client.56633 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi016-74285-57"}]: dispatch 2024-03-26T04:23:51.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:51 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-43"}]': finished 2024-03-26T04:23:51.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:51 smithi016 ceph-mon[29101]: osdmap e306: 8 total, 8 up, 8 in 2024-03-26T04:23:51.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:51 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1963457478' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi016-74285-57"}]: dispatch 2024-03-26T04:23:51.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:51 smithi016 ceph-mon[29101]: from='client.56633 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi016-74285-57"}]: dispatch 2024-03-26T04:23:51.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:51 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-43"}]': finished 2024-03-26T04:23:51.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:51 smithi016 ceph-mon[37215]: osdmap e306: 8 total, 8 up, 8 in 2024-03-26T04:23:51.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:51 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1963457478' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi016-74285-57"}]: dispatch 2024-03-26T04:23:51.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:51 smithi016 ceph-mon[37215]: from='client.56633 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi016-74285-57"}]: dispatch 2024-03-26T04:23:52.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:23:52 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:23:52.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:52 smithi183 ceph-mon[33745]: pgmap v419: 292 pgs: 292 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:23:52.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:52 smithi183 ceph-mon[33745]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:23:52.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:52 smithi183 ceph-mon[33745]: from='client.56633 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi016-74285-57"}]': finished 2024-03-26T04:23:52.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:52 smithi183 ceph-mon[33745]: osdmap e307: 8 total, 8 up, 8 in 2024-03-26T04:23:52.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:52 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1963457478' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi016-74285-57"}]: dispatch 2024-03-26T04:23:52.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:52 smithi183 ceph-mon[33745]: from='client.56633 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi016-74285-57"}]: dispatch 2024-03-26T04:23:52.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:52 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1", "id": [3, 2]}]: dispatch 2024-03-26T04:23:52.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:52 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.13", "id": [3, 1]}]: dispatch 2024-03-26T04:23:52.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:52 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.14", "id": [0, 1]}]: dispatch 2024-03-26T04:23:52.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:52 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1b", "id": [3, 1]}]: dispatch 2024-03-26T04:23:52.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:52 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1d", "id": [3, 7]}]: dispatch 2024-03-26T04:23:52.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:52 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1", "id": [3, 2]}]: dispatch 2024-03-26T04:23:52.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:52 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.13", "id": [3, 1]}]: dispatch 2024-03-26T04:23:52.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:52 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.14", "id": [0, 1]}]: dispatch 2024-03-26T04:23:52.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:52 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1b", "id": [3, 1]}]: dispatch 2024-03-26T04:23:52.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:52 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1d", "id": [3, 7]}]: dispatch 2024-03-26T04:23:52.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:52 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:23:52.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:52 smithi183 ceph-mon[33745]: from='client.56633 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi016-74285-57"}]': finished 2024-03-26T04:23:52.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:52 smithi016 ceph-mon[29101]: pgmap v419: 292 pgs: 292 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:23:52.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:52 smithi016 ceph-mon[29101]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:23:52.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:52 smithi016 ceph-mon[29101]: from='client.56633 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi016-74285-57"}]': finished 2024-03-26T04:23:52.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:52 smithi016 ceph-mon[29101]: osdmap e307: 8 total, 8 up, 8 in 2024-03-26T04:23:52.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:52 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1963457478' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi016-74285-57"}]: dispatch 2024-03-26T04:23:52.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:52 smithi016 ceph-mon[29101]: from='client.56633 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi016-74285-57"}]: dispatch 2024-03-26T04:23:52.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:52 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1", "id": [3, 2]}]: dispatch 2024-03-26T04:23:52.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:52 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.13", "id": [3, 1]}]: dispatch 2024-03-26T04:23:52.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:52 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.14", "id": [0, 1]}]: dispatch 2024-03-26T04:23:52.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:52 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1b", "id": [3, 1]}]: dispatch 2024-03-26T04:23:52.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:52 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1d", "id": [3, 7]}]: dispatch 2024-03-26T04:23:52.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:52 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1", "id": [3, 2]}]: dispatch 2024-03-26T04:23:52.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:52 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.13", "id": [3, 1]}]: dispatch 2024-03-26T04:23:52.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:52 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.14", "id": [0, 1]}]: dispatch 2024-03-26T04:23:52.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:52 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1b", "id": [3, 1]}]: dispatch 2024-03-26T04:23:52.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:52 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1d", "id": [3, 7]}]: dispatch 2024-03-26T04:23:52.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:52 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:23:52.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:52 smithi016 ceph-mon[29101]: from='client.56633 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi016-74285-57"}]': finished 2024-03-26T04:23:52.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:52 smithi016 ceph-mon[37215]: pgmap v419: 292 pgs: 292 active+clean; 8.4 MiB data, 617 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:23:52.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:52 smithi016 ceph-mon[37215]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:23:52.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:52 smithi016 ceph-mon[37215]: from='client.56633 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushPP_smithi016-74285-57"}]': finished 2024-03-26T04:23:52.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:52 smithi016 ceph-mon[37215]: osdmap e307: 8 total, 8 up, 8 in 2024-03-26T04:23:52.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:52 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1963457478' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi016-74285-57"}]: dispatch 2024-03-26T04:23:52.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:52 smithi016 ceph-mon[37215]: from='client.56633 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi016-74285-57"}]: dispatch 2024-03-26T04:23:52.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:52 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1", "id": [3, 2]}]: dispatch 2024-03-26T04:23:52.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:52 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.13", "id": [3, 1]}]: dispatch 2024-03-26T04:23:52.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:52 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.14", "id": [0, 1]}]: dispatch 2024-03-26T04:23:52.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:52 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1b", "id": [3, 1]}]: dispatch 2024-03-26T04:23:52.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:52 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1d", "id": [3, 7]}]: dispatch 2024-03-26T04:23:52.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:52 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1", "id": [3, 2]}]: dispatch 2024-03-26T04:23:52.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:52 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.13", "id": [3, 1]}]: dispatch 2024-03-26T04:23:52.666 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:52 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.14", "id": [0, 1]}]: dispatch 2024-03-26T04:23:52.666 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:52 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1b", "id": [3, 1]}]: dispatch 2024-03-26T04:23:52.666 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:52 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1d", "id": [3, 7]}]: dispatch 2024-03-26T04:23:52.666 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:52 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:23:52.666 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:52 smithi016 ceph-mon[37215]: from='client.56633 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushPP_smithi016-74285-57"}]': finished 2024-03-26T04:23:53.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:53 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1", "id": [3, 2]}]': finished 2024-03-26T04:23:53.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:53 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.13", "id": [3, 1]}]': finished 2024-03-26T04:23:53.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:53 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.14", "id": [0, 1]}]': finished 2024-03-26T04:23:53.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:53 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1b", "id": [3, 1]}]': finished 2024-03-26T04:23:53.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:53 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1d", "id": [3, 7]}]': finished 2024-03-26T04:23:53.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:53 smithi183 ceph-mon[33745]: osdmap e308: 8 total, 8 up, 8 in 2024-03-26T04:23:53.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:53 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:23:53.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:53 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:23:53.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:53 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1048608882' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi016-74285-58"}]: dispatch 2024-03-26T04:23:53.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:53 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1048608882' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi016-74285-58"}]: dispatch 2024-03-26T04:23:53.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:53 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1048608882' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi016-74285-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:53.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:53 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:23:53.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:53 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:23:53.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:53 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1048608882' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi016-74285-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:23:53.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:53 smithi183 ceph-mon[33745]: osdmap e309: 8 total, 8 up, 8 in 2024-03-26T04:23:53.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:53 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1048608882' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi016-74285-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi016-74285-58"}]: dispatch 2024-03-26T04:23:53.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:53 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1", "id": [3, 2]}]': finished 2024-03-26T04:23:53.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:53 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.13", "id": [3, 1]}]': finished 2024-03-26T04:23:53.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:53 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.14", "id": [0, 1]}]': finished 2024-03-26T04:23:53.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:53 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1b", "id": [3, 1]}]': finished 2024-03-26T04:23:53.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:53 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1d", "id": [3, 7]}]': finished 2024-03-26T04:23:53.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:53 smithi016 ceph-mon[29101]: osdmap e308: 8 total, 8 up, 8 in 2024-03-26T04:23:53.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:53 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:23:53.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:53 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:23:53.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:53 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1048608882' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi016-74285-58"}]: dispatch 2024-03-26T04:23:53.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:53 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1048608882' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi016-74285-58"}]: dispatch 2024-03-26T04:23:53.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:53 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1048608882' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi016-74285-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:53.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:53 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:23:53.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:53 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:23:53.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:53 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1048608882' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi016-74285-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:23:53.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:53 smithi016 ceph-mon[29101]: osdmap e309: 8 total, 8 up, 8 in 2024-03-26T04:23:53.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:53 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1048608882' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi016-74285-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi016-74285-58"}]: dispatch 2024-03-26T04:23:53.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:53 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1", "id": [3, 2]}]': finished 2024-03-26T04:23:53.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:53 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.13", "id": [3, 1]}]': finished 2024-03-26T04:23:53.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:53 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.14", "id": [0, 1]}]': finished 2024-03-26T04:23:53.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:53 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1b", "id": [3, 1]}]': finished 2024-03-26T04:23:53.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:53 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "39.1d", "id": [3, 7]}]': finished 2024-03-26T04:23:53.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:53 smithi016 ceph-mon[37215]: osdmap e308: 8 total, 8 up, 8 in 2024-03-26T04:23:53.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:53 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:23:53.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:53 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-45","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:23:53.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:53 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1048608882' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi016-74285-58"}]: dispatch 2024-03-26T04:23:53.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:53 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1048608882' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi016-74285-58"}]: dispatch 2024-03-26T04:23:53.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:53 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1048608882' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi016-74285-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:23:53.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:53 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:23:53.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:53 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-45","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:23:53.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:53 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1048608882' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-FlushAsyncPP_smithi016-74285-58", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:23:53.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:53 smithi016 ceph-mon[37215]: osdmap e309: 8 total, 8 up, 8 in 2024-03-26T04:23:53.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:53 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1048608882' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi016-74285-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi016-74285-58"}]: dispatch 2024-03-26T04:23:54.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:54 smithi183 ceph-mon[33745]: pgmap v422: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:23:54.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:54 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:23:54.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:54 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:23:54.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:54 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-45", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:23:54.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:54 smithi016 ceph-mon[29101]: pgmap v422: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:23:54.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:54 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:23:54.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:54 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:23:54.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:54 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-45", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:23:54.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:54 smithi016 ceph-mon[37215]: pgmap v422: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:23:54.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:54 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:23:54.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:54 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-45", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:23:54.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:54 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-45", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:23:55.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:55 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T04:23:55.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:55 smithi183 ceph-mon[33745]: osdmap e310: 8 total, 8 up, 8 in 2024-03-26T04:23:55.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:55 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T04:23:55.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:55 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1048608882' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi016-74285-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi016-74285-58"}]': finished 2024-03-26T04:23:55.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:55 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-45","var": "hit_set_count","val": "3"}]': finished 2024-03-26T04:23:55.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:55 smithi183 ceph-mon[33745]: osdmap e311: 8 total, 8 up, 8 in 2024-03-26T04:23:55.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:55 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T04:23:55.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:55 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T04:23:55.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:55 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T04:23:55.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:55 smithi016 ceph-mon[29101]: osdmap e310: 8 total, 8 up, 8 in 2024-03-26T04:23:55.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:55 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T04:23:55.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:55 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1048608882' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi016-74285-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi016-74285-58"}]': finished 2024-03-26T04:23:55.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:55 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-45","var": "hit_set_count","val": "3"}]': finished 2024-03-26T04:23:55.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:55 smithi016 ceph-mon[29101]: osdmap e311: 8 total, 8 up, 8 in 2024-03-26T04:23:55.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:55 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T04:23:55.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:55 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T04:23:55.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:55 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T04:23:55.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:55 smithi016 ceph-mon[37215]: osdmap e310: 8 total, 8 up, 8 in 2024-03-26T04:23:55.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:55 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-45","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T04:23:55.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:55 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1048608882' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "FlushAsyncPP_smithi016-74285-58", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-FlushAsyncPP_smithi016-74285-58"}]': finished 2024-03-26T04:23:55.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:55 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-45","var": "hit_set_count","val": "3"}]': finished 2024-03-26T04:23:55.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:55 smithi016 ceph-mon[37215]: osdmap e311: 8 total, 8 up, 8 in 2024-03-26T04:23:55.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:55 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T04:23:55.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:55 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-45","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T04:23:56.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:56 smithi183 ceph-mon[33745]: pgmap v425: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:23:56.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:56 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-45","var": "hit_set_period","val": "3"}]': finished 2024-03-26T04:23:56.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:56 smithi183 ceph-mon[33745]: osdmap e312: 8 total, 8 up, 8 in 2024-03-26T04:23:56.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:56 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:23:56.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:56 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:23:56.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:56 smithi016 ceph-mon[37215]: pgmap v425: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:23:56.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:56 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-45","var": "hit_set_period","val": "3"}]': finished 2024-03-26T04:23:56.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:56 smithi016 ceph-mon[37215]: osdmap e312: 8 total, 8 up, 8 in 2024-03-26T04:23:56.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:56 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:23:56.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:56 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:23:56.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:56 smithi016 ceph-mon[29101]: pgmap v425: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:23:56.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:56 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-45","var": "hit_set_period","val": "3"}]': finished 2024-03-26T04:23:56.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:56 smithi016 ceph-mon[29101]: osdmap e312: 8 total, 8 up, 8 in 2024-03-26T04:23:56.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:56 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:23:56.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:56 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-45","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:23:58.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:58 smithi183 ceph-mon[33745]: pgmap v428: 300 pgs: 30 unknown, 270 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 255 B/s wr, 0 op/s 2024-03-26T04:23:58.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:58 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-45","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T04:23:58.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:58 smithi183 ceph-mon[33745]: osdmap e313: 8 total, 8 up, 8 in 2024-03-26T04:23:58.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1048608882' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi016-74285-58"}]: dispatch 2024-03-26T04:23:58.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T04:23:58.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:58 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T04:23:58.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:58 smithi183 ceph-mon[33745]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:23:58.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:58 smithi016 ceph-mon[37215]: pgmap v428: 300 pgs: 30 unknown, 270 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 255 B/s wr, 0 op/s 2024-03-26T04:23:58.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:58 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-45","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T04:23:58.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:58 smithi016 ceph-mon[37215]: osdmap e313: 8 total, 8 up, 8 in 2024-03-26T04:23:58.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1048608882' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi016-74285-58"}]: dispatch 2024-03-26T04:23:58.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T04:23:58.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:58 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T04:23:58.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:58 smithi016 ceph-mon[37215]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:23:58.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:58 smithi016 ceph-mon[29101]: pgmap v428: 300 pgs: 30 unknown, 270 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 255 B/s wr, 0 op/s 2024-03-26T04:23:58.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:58 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-45","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T04:23:58.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:58 smithi016 ceph-mon[29101]: osdmap e313: 8 total, 8 up, 8 in 2024-03-26T04:23:58.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1048608882' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi016-74285-58"}]: dispatch 2024-03-26T04:23:58.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T04:23:58.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:58 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-45","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T04:23:58.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:58 smithi016 ceph-mon[29101]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:23:59.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:23:58 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:23:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:23:59.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:59 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1048608882' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi016-74285-58"}]': finished 2024-03-26T04:23:59.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:59 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-26T04:23:59.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:59 smithi183 ceph-mon[33745]: osdmap e314: 8 total, 8 up, 8 in 2024-03-26T04:23:59.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:59 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1048608882' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi016-74285-58"}]: dispatch 2024-03-26T04:23:59.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:59 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1048608882' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi016-74285-58"}]': finished 2024-03-26T04:23:59.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:23:59 smithi183 ceph-mon[33745]: osdmap e315: 8 total, 8 up, 8 in 2024-03-26T04:23:59.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:59 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1048608882' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi016-74285-58"}]': finished 2024-03-26T04:23:59.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:59 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-26T04:23:59.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:59 smithi016 ceph-mon[37215]: osdmap e314: 8 total, 8 up, 8 in 2024-03-26T04:23:59.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:59 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1048608882' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi016-74285-58"}]: dispatch 2024-03-26T04:23:59.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:59 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1048608882' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi016-74285-58"}]': finished 2024-03-26T04:23:59.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:23:59 smithi016 ceph-mon[37215]: osdmap e315: 8 total, 8 up, 8 in 2024-03-26T04:23:59.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:59 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1048608882' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-FlushAsyncPP_smithi016-74285-58"}]': finished 2024-03-26T04:23:59.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:59 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-45","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-26T04:23:59.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:59 smithi016 ceph-mon[29101]: osdmap e314: 8 total, 8 up, 8 in 2024-03-26T04:23:59.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:59 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1048608882' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi016-74285-58"}]: dispatch 2024-03-26T04:23:59.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:59 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1048608882' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"FlushAsyncPP_smithi016-74285-58"}]': finished 2024-03-26T04:23:59.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:23:59 smithi016 ceph-mon[29101]: osdmap e315: 8 total, 8 up, 8 in 2024-03-26T04:24:00.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:00 smithi183 ceph-mon[33745]: pgmap v431: 292 pgs: 292 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:24:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:00 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3064040512' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi016-74285-59"}]: dispatch 2024-03-26T04:24:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:00 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3064040512' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi016-74285-59"}]: dispatch 2024-03-26T04:24:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:00 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3064040512' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi016-74285-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:24:00.564 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:00 smithi016 ceph-mon[37215]: pgmap v431: 292 pgs: 292 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:24:00.564 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:00 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3064040512' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi016-74285-59"}]: dispatch 2024-03-26T04:24:00.564 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:00 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3064040512' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi016-74285-59"}]: dispatch 2024-03-26T04:24:00.564 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:00 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3064040512' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi016-74285-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:24:00.564 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:00 smithi016 ceph-mon[29101]: pgmap v431: 292 pgs: 292 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:24:00.564 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:00 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3064040512' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi016-74285-59"}]: dispatch 2024-03-26T04:24:00.565 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:00 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3064040512' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi016-74285-59"}]: dispatch 2024-03-26T04:24:00.565 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:00 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3064040512' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi016-74285-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:24:01.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:01 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3064040512' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi016-74285-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:24:01.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:01 smithi183 ceph-mon[33745]: osdmap e316: 8 total, 8 up, 8 in 2024-03-26T04:24:01.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:01 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3064040512' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi016-74285-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi016-74285-59"}]: dispatch 2024-03-26T04:24:01.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:01 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3064040512' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi016-74285-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:24:01.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:01 smithi016 ceph-mon[37215]: osdmap e316: 8 total, 8 up, 8 in 2024-03-26T04:24:01.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:01 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3064040512' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi016-74285-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi016-74285-59"}]: dispatch 2024-03-26T04:24:01.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:01 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3064040512' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-RoundTripWriteFullPP_smithi016-74285-59", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:24:01.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:01 smithi016 ceph-mon[29101]: osdmap e316: 8 total, 8 up, 8 in 2024-03-26T04:24:01.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:01 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3064040512' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi016-74285-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi016-74285-59"}]: dispatch 2024-03-26T04:24:02.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:02 smithi183 ceph-mon[33745]: pgmap v434: 292 pgs: 292 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:24:02.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:02 smithi183 ceph-mon[33745]: osdmap e317: 8 total, 8 up, 8 in 2024-03-26T04:24:02.542 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:24:02 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:24:02.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:02 smithi016 ceph-mon[37215]: pgmap v434: 292 pgs: 292 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:24:02.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:02 smithi016 ceph-mon[37215]: osdmap e317: 8 total, 8 up, 8 in 2024-03-26T04:24:02.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:02 smithi016 ceph-mon[29101]: pgmap v434: 292 pgs: 292 active+clean; 8.4 MiB data, 622 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:24:02.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:02 smithi016 ceph-mon[29101]: osdmap e317: 8 total, 8 up, 8 in 2024-03-26T04:24:03.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:03 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3064040512' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi016-74285-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi016-74285-59"}]': finished 2024-03-26T04:24:03.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:03 smithi183 ceph-mon[33745]: osdmap e318: 8 total, 8 up, 8 in 2024-03-26T04:24:03.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:03 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:24:03.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:03 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3064040512' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi016-74285-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi016-74285-59"}]': finished 2024-03-26T04:24:03.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:03 smithi016 ceph-mon[37215]: osdmap e318: 8 total, 8 up, 8 in 2024-03-26T04:24:03.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:03 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:24:03.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:03 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3064040512' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "RoundTripWriteFullPP_smithi016-74285-59", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-RoundTripWriteFullPP_smithi016-74285-59"}]': finished 2024-03-26T04:24:03.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:03 smithi016 ceph-mon[29101]: osdmap e318: 8 total, 8 up, 8 in 2024-03-26T04:24:03.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:03 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:24:04.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:04 smithi183 ceph-mon[33745]: pgmap v437: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T04:24:04.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:04 smithi183 ceph-mon[33745]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:24:04.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:04 smithi183 ceph-mon[33745]: osdmap e319: 8 total, 8 up, 8 in 2024-03-26T04:24:04.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:04 smithi016 ceph-mon[29101]: pgmap v437: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T04:24:04.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:04 smithi016 ceph-mon[29101]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:24:04.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:04 smithi016 ceph-mon[29101]: osdmap e319: 8 total, 8 up, 8 in 2024-03-26T04:24:04.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:04 smithi016 ceph-mon[37215]: pgmap v437: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T04:24:04.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:04 smithi016 ceph-mon[37215]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:24:04.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:04 smithi016 ceph-mon[37215]: osdmap e319: 8 total, 8 up, 8 in 2024-03-26T04:24:05.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:05 smithi183 ceph-mon[33745]: osdmap e320: 8 total, 8 up, 8 in 2024-03-26T04:24:05.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:05 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3064040512' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi016-74285-59"}]: dispatch 2024-03-26T04:24:05.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:05 smithi016 ceph-mon[29101]: osdmap e320: 8 total, 8 up, 8 in 2024-03-26T04:24:05.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:05 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3064040512' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi016-74285-59"}]: dispatch 2024-03-26T04:24:05.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:05 smithi016 ceph-mon[37215]: osdmap e320: 8 total, 8 up, 8 in 2024-03-26T04:24:05.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:05 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3064040512' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi016-74285-59"}]: dispatch 2024-03-26T04:24:06.274 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:7462ddf6::o.RoundTripAppendPP (3020 ms) 2024-03-26T04:24:06.275 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-03-26T04:24:06.275 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3193 ms) 2024-03-26T04:24:06.275 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-03-26T04:24:06.275 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (2756 ms) 2024-03-26T04:24:06.275 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-03-26T04:24:06.275 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3006 ms) 2024-03-26T04:24:06.275 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-03-26T04:24:06.275 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: setting pool EIO 2024-03-26T04:24:06.275 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: max_success 99, min_failed 100 2024-03-26T04:24:06.275 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (4014 ms) 2024-03-26T04:24:06.275 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-03-26T04:24:06.275 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (3021 ms) 2024-03-26T04:24:06.276 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (130597 ms total) 2024-03-26T04:24:06.276 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: 2024-03-26T04:24:06.276 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-03-26T04:24:06.276 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-03-26T04:24:06.276 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3010 ms) 2024-03-26T04:24:06.276 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-03-26T04:24:06.276 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (9035 ms) 2024-03-26T04:24:06.276 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-03-26T04:24:06.276 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (14959 ms) 2024-03-26T04:24:06.276 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-03-26T04:24:06.276 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3267 ms) 2024-03-26T04:24:06.276 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (30272 ms total) 2024-03-26T04:24:06.276 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: 2024-03-26T04:24:06.276 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-03-26T04:24:06.276 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-03-26T04:24:06.277 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (2749 ms) 2024-03-26T04:24:06.277 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (2749 ms total) 2024-03-26T04:24:06.277 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: 2024-03-26T04:24:06.277 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-03-26T04:24:06.277 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-03-26T04:24:06.277 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (13956 ms) 2024-03-26T04:24:06.277 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-03-26T04:24:06.277 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (6998 ms) 2024-03-26T04:24:06.277 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-03-26T04:24:06.277 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (7031 ms) 2024-03-26T04:24:06.277 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-03-26T04:24:06.277 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7015 ms) 2024-03-26T04:24:06.277 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-03-26T04:24:06.277 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3259 ms) 2024-03-26T04:24:06.278 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-03-26T04:24:06.278 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (6761 ms) 2024-03-26T04:24:06.278 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-03-26T04:24:06.278 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (6940 ms) 2024-03-26T04:24:06.278 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-03-26T04:24:06.278 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (7022 ms) 2024-03-26T04:24:06.278 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-03-26T04:24:06.278 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7028 ms) 2024-03-26T04:24:06.278 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-03-26T04:24:06.278 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (7014 ms) 2024-03-26T04:24:06.278 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-03-26T04:24:06.278 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7034 ms) 2024-03-26T04:24:06.278 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-03-26T04:24:06.279 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (7012 ms) 2024-03-26T04:24:06.279 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-03-26T04:24:06.279 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (7054 ms) 2024-03-26T04:24:06.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:06 smithi183 ceph-mon[33745]: pgmap v440: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T04:24:06.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:06 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3064040512' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi016-74285-59"}]': finished 2024-03-26T04:24:06.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:06 smithi183 ceph-mon[33745]: osdmap e321: 8 total, 8 up, 8 in 2024-03-26T04:24:06.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:06 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3064040512' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi016-74285-59"}]: dispatch 2024-03-26T04:24:06.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:06 smithi016 ceph-mon[29101]: pgmap v440: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T04:24:06.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:06 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3064040512' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi016-74285-59"}]': finished 2024-03-26T04:24:06.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:06 smithi016 ceph-mon[29101]: osdmap e321: 8 total, 8 up, 8 in 2024-03-26T04:24:06.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:06 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3064040512' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi016-74285-59"}]: dispatch 2024-03-26T04:24:06.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:06 smithi016 ceph-mon[37215]: pgmap v440: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T04:24:06.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:06 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3064040512' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-RoundTripWriteFullPP_smithi016-74285-59"}]': finished 2024-03-26T04:24:06.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:06 smithi016 ceph-mon[37215]: osdmap e321: 8 total, 8 up, 8 in 2024-03-26T04:24:06.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:06 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3064040512' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi016-74285-59"}]: dispatch 2024-03-26T04:24:07.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:07 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3064040512' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi016-74285-59"}]': finished 2024-03-26T04:24:07.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:07 smithi183 ceph-mon[33745]: osdmap e322: 8 total, 8 up, 8 in 2024-03-26T04:24:07.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:07 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:24:07.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:07 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3064040512' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi016-74285-59"}]': finished 2024-03-26T04:24:07.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:07 smithi016 ceph-mon[29101]: osdmap e322: 8 total, 8 up, 8 in 2024-03-26T04:24:07.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:07 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:24:07.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:07 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3064040512' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"RoundTripWriteFullPP_smithi016-74285-59"}]': finished 2024-03-26T04:24:07.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:07 smithi016 ceph-mon[37215]: osdmap e322: 8 total, 8 up, 8 in 2024-03-26T04:24:07.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:07 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:24:08.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:08 smithi183 ceph-mon[33745]: pgmap v443: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s wr, 1 op/s 2024-03-26T04:24:08.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:08 smithi183 ceph-mon[33745]: osdmap e323: 8 total, 8 up, 8 in 2024-03-26T04:24:08.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:08 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/213540531' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi016-74285-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:24:08.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:08 smithi183 ceph-mon[33745]: from='client.56651 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi016-74285-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:24:08.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:08 smithi016 ceph-mon[29101]: pgmap v443: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s wr, 1 op/s 2024-03-26T04:24:08.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:08 smithi016 ceph-mon[29101]: osdmap e323: 8 total, 8 up, 8 in 2024-03-26T04:24:08.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:08 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/213540531' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi016-74285-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:24:08.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:08 smithi016 ceph-mon[29101]: from='client.56651 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi016-74285-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:24:08.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:08 smithi016 ceph-mon[37215]: pgmap v443: 292 pgs: 292 active+clean; 8.4 MiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s wr, 1 op/s 2024-03-26T04:24:08.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:08 smithi016 ceph-mon[37215]: osdmap e323: 8 total, 8 up, 8 in 2024-03-26T04:24:08.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:08 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/213540531' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi016-74285-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:24:08.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:08 smithi016 ceph-mon[37215]: from='client.56651 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi016-74285-60","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:24:09.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:24:08 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:24:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:24:09.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:09 smithi016 ceph-mon[37215]: from='client.56651 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi016-74285-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:24:09.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:09 smithi016 ceph-mon[37215]: osdmap e324: 8 total, 8 up, 8 in 2024-03-26T04:24:09.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:09 smithi016 ceph-mon[29101]: from='client.56651 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi016-74285-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:24:09.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:09 smithi016 ceph-mon[29101]: osdmap e324: 8 total, 8 up, 8 in 2024-03-26T04:24:09.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:09 smithi183 ceph-mon[33745]: from='client.56651 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "RoundTripWriteFullPP2_smithi016-74285-60","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:24:09.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:09 smithi183 ceph-mon[33745]: osdmap e324: 8 total, 8 up, 8 in 2024-03-26T04:24:10.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:10 smithi016 ceph-mon[37215]: pgmap v446: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T04:24:10.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:10 smithi016 ceph-mon[37215]: osdmap e325: 8 total, 8 up, 8 in 2024-03-26T04:24:10.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:10 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1390855576' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi016-74285-61"}]: dispatch 2024-03-26T04:24:10.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:10 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1390855576' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi016-74285-61"}]: dispatch 2024-03-26T04:24:10.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:10 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1390855576' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi016-74285-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:24:10.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:10 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:24:10.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:10 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:24:10.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:10 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-45"}]: dispatch 2024-03-26T04:24:10.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:10 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-45"}]: dispatch 2024-03-26T04:24:10.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:10 smithi016 ceph-mon[29101]: pgmap v446: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T04:24:10.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:10 smithi016 ceph-mon[29101]: osdmap e325: 8 total, 8 up, 8 in 2024-03-26T04:24:10.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:10 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1390855576' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi016-74285-61"}]: dispatch 2024-03-26T04:24:10.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:10 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1390855576' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi016-74285-61"}]: dispatch 2024-03-26T04:24:10.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:10 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1390855576' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi016-74285-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:24:10.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:10 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:24:10.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:10 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:24:10.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:10 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-45"}]: dispatch 2024-03-26T04:24:10.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:10 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-45"}]: dispatch 2024-03-26T04:24:10.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:10 smithi183 ceph-mon[33745]: pgmap v446: 324 pgs: 32 unknown, 292 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T04:24:10.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:10 smithi183 ceph-mon[33745]: osdmap e325: 8 total, 8 up, 8 in 2024-03-26T04:24:10.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:10 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1390855576' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi016-74285-61"}]: dispatch 2024-03-26T04:24:10.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:10 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1390855576' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi016-74285-61"}]: dispatch 2024-03-26T04:24:10.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:10 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1390855576' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi016-74285-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:24:10.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:10 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:24:10.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:10 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:24:10.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:10 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-45"}]: dispatch 2024-03-26T04:24:10.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:10 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-45"}]: dispatch 2024-03-26T04:24:11.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:11 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1390855576' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi016-74285-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:24:11.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:11 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-45"}]': finished 2024-03-26T04:24:11.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:11 smithi016 ceph-mon[37215]: osdmap e326: 8 total, 8 up, 8 in 2024-03-26T04:24:11.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:11 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1390855576' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi016-74285-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi016-74285-61"}]: dispatch 2024-03-26T04:24:11.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:11 smithi016 ceph-mon[37215]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:24:11.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:11 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1390855576' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi016-74285-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:24:11.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:11 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-45"}]': finished 2024-03-26T04:24:11.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:11 smithi016 ceph-mon[29101]: osdmap e326: 8 total, 8 up, 8 in 2024-03-26T04:24:11.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:11 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1390855576' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi016-74285-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi016-74285-61"}]: dispatch 2024-03-26T04:24:11.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:11 smithi016 ceph-mon[29101]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:24:11.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:11 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1390855576' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPP_smithi016-74285-61", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:24:11.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:11 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-45"}]': finished 2024-03-26T04:24:11.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:11 smithi183 ceph-mon[33745]: osdmap e326: 8 total, 8 up, 8 in 2024-03-26T04:24:11.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:11 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1390855576' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi016-74285-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi016-74285-61"}]: dispatch 2024-03-26T04:24:11.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:11 smithi183 ceph-mon[33745]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:24:12.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:12 smithi183 ceph-mon[33745]: pgmap v449: 292 pgs: 292 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:24:12.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:12 smithi183 ceph-mon[33745]: osdmap e327: 8 total, 8 up, 8 in 2024-03-26T04:24:12.542 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:24:12 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:24:12.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:12 smithi016 ceph-mon[29101]: pgmap v449: 292 pgs: 292 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:24:12.671 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:12 smithi016 ceph-mon[29101]: osdmap e327: 8 total, 8 up, 8 in 2024-03-26T04:24:12.671 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:12 smithi016 ceph-mon[37215]: pgmap v449: 292 pgs: 292 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:24:12.672 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:12 smithi016 ceph-mon[37215]: osdmap e327: 8 total, 8 up, 8 in 2024-03-26T04:24:13.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:13 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:24:13.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:13 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1390855576' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi016-74285-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi016-74285-61"}]': finished 2024-03-26T04:24:13.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:13 smithi016 ceph-mon[29101]: osdmap e328: 8 total, 8 up, 8 in 2024-03-26T04:24:13.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:13 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:24:13.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:13 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:24:13.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:13 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:24:13.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:13 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1390855576' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi016-74285-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi016-74285-61"}]': finished 2024-03-26T04:24:13.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:13 smithi016 ceph-mon[37215]: osdmap e328: 8 total, 8 up, 8 in 2024-03-26T04:24:13.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:13 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:24:13.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:13 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:24:13.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:13 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:24:13.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:13 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1390855576' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPP_smithi016-74285-61", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPP_smithi016-74285-61"}]': finished 2024-03-26T04:24:13.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:13 smithi183 ceph-mon[33745]: osdmap e328: 8 total, 8 up, 8 in 2024-03-26T04:24:13.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:13 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:24:13.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:13 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-47","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:24:14.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:14 smithi016 ceph-mon[37215]: pgmap v452: 300 pgs: 40 unknown, 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:24:14.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:14 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:24:14.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:14 smithi016 ceph-mon[37215]: osdmap e329: 8 total, 8 up, 8 in 2024-03-26T04:24:14.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:14 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:24:14.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:14 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:24:14.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:14 smithi016 ceph-mon[29101]: pgmap v452: 300 pgs: 40 unknown, 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:24:14.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:14 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:24:14.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:14 smithi016 ceph-mon[29101]: osdmap e329: 8 total, 8 up, 8 in 2024-03-26T04:24:14.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:14 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:24:14.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:14 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:24:14.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:14 smithi183 ceph-mon[33745]: pgmap v452: 300 pgs: 40 unknown, 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:24:14.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:14 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-47","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:24:14.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:14 smithi183 ceph-mon[33745]: osdmap e329: 8 total, 8 up, 8 in 2024-03-26T04:24:14.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:14 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:24:14.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:14 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-47", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:24:15.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:15 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-47", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:24:15.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:15 smithi016 ceph-mon[37215]: osdmap e330: 8 total, 8 up, 8 in 2024-03-26T04:24:15.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:15 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1390855576' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi016-74285-61"}]: dispatch 2024-03-26T04:24:15.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:15 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-47"}]: dispatch 2024-03-26T04:24:15.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:15 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-47"}]: dispatch 2024-03-26T04:24:15.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:15 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-47", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:24:15.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:15 smithi016 ceph-mon[29101]: osdmap e330: 8 total, 8 up, 8 in 2024-03-26T04:24:15.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:15 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1390855576' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi016-74285-61"}]: dispatch 2024-03-26T04:24:15.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:15 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-47"}]: dispatch 2024-03-26T04:24:15.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:15 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-47"}]: dispatch 2024-03-26T04:24:15.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:15 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-47", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:24:15.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:15 smithi183 ceph-mon[33745]: osdmap e330: 8 total, 8 up, 8 in 2024-03-26T04:24:15.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:15 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1390855576' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi016-74285-61"}]: dispatch 2024-03-26T04:24:15.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:15 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-47"}]: dispatch 2024-03-26T04:24:15.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:15 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-47"}]: dispatch 2024-03-26T04:24:16.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:16 smithi016 ceph-mon[37215]: pgmap v455: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:24:16.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:16 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1390855576' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi016-74285-61"}]': finished 2024-03-26T04:24:16.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:16 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-47"}]': finished 2024-03-26T04:24:16.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:16 smithi016 ceph-mon[37215]: osdmap e331: 8 total, 8 up, 8 in 2024-03-26T04:24:16.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:16 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1390855576' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi016-74285-61"}]: dispatch 2024-03-26T04:24:16.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:16 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-47", "mode": "writeback"}]: dispatch 2024-03-26T04:24:16.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:16 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-47", "mode": "writeback"}]: dispatch 2024-03-26T04:24:16.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:16 smithi016 ceph-mon[29101]: pgmap v455: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:24:16.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:16 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1390855576' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi016-74285-61"}]': finished 2024-03-26T04:24:16.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:16 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-47"}]': finished 2024-03-26T04:24:16.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:16 smithi016 ceph-mon[29101]: osdmap e331: 8 total, 8 up, 8 in 2024-03-26T04:24:16.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:16 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1390855576' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi016-74285-61"}]: dispatch 2024-03-26T04:24:16.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:16 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-47", "mode": "writeback"}]: dispatch 2024-03-26T04:24:16.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:16 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-47", "mode": "writeback"}]: dispatch 2024-03-26T04:24:16.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:16 smithi183 ceph-mon[33745]: pgmap v455: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 632 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:24:16.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:16 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1390855576' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPP_smithi016-74285-61"}]': finished 2024-03-26T04:24:16.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:16 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-47"}]': finished 2024-03-26T04:24:16.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:16 smithi183 ceph-mon[33745]: osdmap e331: 8 total, 8 up, 8 in 2024-03-26T04:24:16.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:16 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1390855576' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi016-74285-61"}]: dispatch 2024-03-26T04:24:16.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:16 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-47", "mode": "writeback"}]: dispatch 2024-03-26T04:24:16.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:16 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-47", "mode": "writeback"}]: dispatch 2024-03-26T04:24:17.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:17 smithi016 ceph-mon[29101]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:24:17.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:17 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1390855576' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi016-74285-61"}]': finished 2024-03-26T04:24:17.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:17 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-47", "mode": "writeback"}]': finished 2024-03-26T04:24:17.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:17 smithi016 ceph-mon[29101]: osdmap e332: 8 total, 8 up, 8 in 2024-03-26T04:24:17.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:17 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:24:17.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:17 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:24:17.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:17 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3721104634' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi016-74285-62"}]: dispatch 2024-03-26T04:24:17.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:17 smithi016 ceph-mon[29101]: from='client.56663 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi016-74285-62"}]: dispatch 2024-03-26T04:24:17.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:17 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3721104634' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi016-74285-62"}]: dispatch 2024-03-26T04:24:17.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:17 smithi016 ceph-mon[29101]: from='client.56663 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi016-74285-62"}]: dispatch 2024-03-26T04:24:17.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:17 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3721104634' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi016-74285-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:24:17.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:17 smithi016 ceph-mon[29101]: from='client.56663 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi016-74285-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:24:17.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:17 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "hit_set_count","val": "2"}]': finished 2024-03-26T04:24:17.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:17 smithi016 ceph-mon[29101]: from='client.56663 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi016-74285-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:24:17.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:17 smithi016 ceph-mon[29101]: osdmap e333: 8 total, 8 up, 8 in 2024-03-26T04:24:17.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:17 smithi016 ceph-mon[37215]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:24:17.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:17 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1390855576' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi016-74285-61"}]': finished 2024-03-26T04:24:17.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:17 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-47", "mode": "writeback"}]': finished 2024-03-26T04:24:17.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:17 smithi016 ceph-mon[37215]: osdmap e332: 8 total, 8 up, 8 in 2024-03-26T04:24:17.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:17 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:24:17.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:17 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:24:17.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:17 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3721104634' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi016-74285-62"}]: dispatch 2024-03-26T04:24:17.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:17 smithi016 ceph-mon[37215]: from='client.56663 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi016-74285-62"}]: dispatch 2024-03-26T04:24:17.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:17 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3721104634' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi016-74285-62"}]: dispatch 2024-03-26T04:24:17.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:17 smithi016 ceph-mon[37215]: from='client.56663 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi016-74285-62"}]: dispatch 2024-03-26T04:24:17.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:17 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3721104634' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi016-74285-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:24:17.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:17 smithi016 ceph-mon[37215]: from='client.56663 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi016-74285-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:24:17.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:17 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "hit_set_count","val": "2"}]': finished 2024-03-26T04:24:17.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:17 smithi016 ceph-mon[37215]: from='client.56663 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi016-74285-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:24:17.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:17 smithi016 ceph-mon[37215]: osdmap e333: 8 total, 8 up, 8 in 2024-03-26T04:24:17.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:17 smithi183 ceph-mon[33745]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:24:17.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:17 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1390855576' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPP_smithi016-74285-61"}]': finished 2024-03-26T04:24:17.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:17 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-47", "mode": "writeback"}]': finished 2024-03-26T04:24:17.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:17 smithi183 ceph-mon[33745]: osdmap e332: 8 total, 8 up, 8 in 2024-03-26T04:24:17.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:17 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:24:17.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:17 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:24:17.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:17 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3721104634' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi016-74285-62"}]: dispatch 2024-03-26T04:24:17.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:17 smithi183 ceph-mon[33745]: from='client.56663 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi016-74285-62"}]: dispatch 2024-03-26T04:24:17.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:17 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3721104634' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi016-74285-62"}]: dispatch 2024-03-26T04:24:17.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:17 smithi183 ceph-mon[33745]: from='client.56663 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi016-74285-62"}]: dispatch 2024-03-26T04:24:17.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:17 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3721104634' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi016-74285-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:24:17.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:17 smithi183 ceph-mon[33745]: from='client.56663 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi016-74285-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:24:17.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:17 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "hit_set_count","val": "2"}]': finished 2024-03-26T04:24:17.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:17 smithi183 ceph-mon[33745]: from='client.56663 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-SimpleStatPPNS_smithi016-74285-62", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:24:17.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:17 smithi183 ceph-mon[33745]: osdmap e333: 8 total, 8 up, 8 in 2024-03-26T04:24:18.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:24:18 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:24:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:24:18.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:18 smithi016 ceph-mon[37215]: pgmap v458: 292 pgs: 18 unknown, 274 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2024-03-26T04:24:18.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:18 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:24:18.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:18 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3721104634' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi016-74285-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi016-74285-62"}]: dispatch 2024-03-26T04:24:18.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:18 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:24:18.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:18 smithi016 ceph-mon[37215]: from='client.56663 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi016-74285-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi016-74285-62"}]: dispatch 2024-03-26T04:24:18.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:18 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:24:18.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:18 smithi016 ceph-mon[29101]: pgmap v458: 292 pgs: 18 unknown, 274 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2024-03-26T04:24:18.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:18 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:24:18.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:18 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3721104634' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi016-74285-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi016-74285-62"}]: dispatch 2024-03-26T04:24:18.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:18 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:24:18.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:18 smithi016 ceph-mon[29101]: from='client.56663 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi016-74285-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi016-74285-62"}]: dispatch 2024-03-26T04:24:18.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:18 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:24:19.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:18 smithi183 ceph-mon[33745]: pgmap v458: 292 pgs: 18 unknown, 274 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2024-03-26T04:24:19.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:18 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:24:19.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:18 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3721104634' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi016-74285-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi016-74285-62"}]: dispatch 2024-03-26T04:24:19.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:18 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:24:19.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:18 smithi183 ceph-mon[33745]: from='client.56663 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi016-74285-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi016-74285-62"}]: dispatch 2024-03-26T04:24:19.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:18 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:24:19.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:19 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:24:19.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:19 smithi016 ceph-mon[29101]: osdmap e334: 8 total, 8 up, 8 in 2024-03-26T04:24:19.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:19 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:24:19.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:19 smithi016 ceph-mon[29101]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:24:19.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:19 smithi016 ceph-mon[29101]: from='client.56663 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi016-74285-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi016-74285-62"}]': finished 2024-03-26T04:24:19.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:19 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T04:24:19.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:19 smithi016 ceph-mon[29101]: osdmap e335: 8 total, 8 up, 8 in 2024-03-26T04:24:19.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:19 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T04:24:19.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:19 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T04:24:19.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:19 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:24:19.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:19 smithi016 ceph-mon[37215]: osdmap e334: 8 total, 8 up, 8 in 2024-03-26T04:24:19.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:19 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:24:19.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:19 smithi016 ceph-mon[37215]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:24:19.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:19 smithi016 ceph-mon[37215]: from='client.56663 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi016-74285-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi016-74285-62"}]': finished 2024-03-26T04:24:19.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:19 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T04:24:19.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:19 smithi016 ceph-mon[37215]: osdmap e335: 8 total, 8 up, 8 in 2024-03-26T04:24:19.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:19 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T04:24:19.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:19 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T04:24:20.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:19 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:24:20.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:19 smithi183 ceph-mon[33745]: osdmap e334: 8 total, 8 up, 8 in 2024-03-26T04:24:20.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:19 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:24:20.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:19 smithi183 ceph-mon[33745]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:24:20.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:19 smithi183 ceph-mon[33745]: from='client.56663 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleStatPPNS_smithi016-74285-62", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleStatPPNS_smithi016-74285-62"}]': finished 2024-03-26T04:24:20.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:19 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T04:24:20.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:19 smithi183 ceph-mon[33745]: osdmap e335: 8 total, 8 up, 8 in 2024-03-26T04:24:20.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:19 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T04:24:20.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:19 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T04:24:20.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:20 smithi016 ceph-mon[29101]: pgmap v461: 292 pgs: 292 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-26T04:24:20.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:20 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T04:24:20.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:20 smithi016 ceph-mon[29101]: osdmap e336: 8 total, 8 up, 8 in 2024-03-26T04:24:20.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:20 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T04:24:20.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:20 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T04:24:20.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:20 smithi016 ceph-mon[37215]: pgmap v461: 292 pgs: 292 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-26T04:24:20.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:20 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T04:24:20.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:20 smithi016 ceph-mon[37215]: osdmap e336: 8 total, 8 up, 8 in 2024-03-26T04:24:20.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:20 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T04:24:20.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:20 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T04:24:21.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:20 smithi183 ceph-mon[33745]: pgmap v461: 292 pgs: 292 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-26T04:24:21.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:20 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T04:24:21.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:20 smithi183 ceph-mon[33745]: osdmap e336: 8 total, 8 up, 8 in 2024-03-26T04:24:21.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:20 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T04:24:21.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:20 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T04:24:22.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:24:22 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:24:22.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:22 smithi183 ceph-mon[33745]: pgmap v464: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.5 KiB/s wr, 5 op/s 2024-03-26T04:24:22.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:22 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-26T04:24:22.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:22 smithi183 ceph-mon[33745]: osdmap e337: 8 total, 8 up, 8 in 2024-03-26T04:24:22.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:22 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3721104634' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi016-74285-62"}]: dispatch 2024-03-26T04:24:22.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:22 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T04:24:22.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:22 smithi183 ceph-mon[33745]: from='client.56663 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi016-74285-62"}]: dispatch 2024-03-26T04:24:22.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:22 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T04:24:22.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:22 smithi183 ceph-mon[33745]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:24:22.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:22 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:24:22.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:22 smithi016 ceph-mon[29101]: pgmap v464: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.5 KiB/s wr, 5 op/s 2024-03-26T04:24:22.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:22 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-26T04:24:22.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:22 smithi016 ceph-mon[29101]: osdmap e337: 8 total, 8 up, 8 in 2024-03-26T04:24:22.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:22 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3721104634' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi016-74285-62"}]: dispatch 2024-03-26T04:24:22.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:22 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T04:24:22.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:22 smithi016 ceph-mon[29101]: from='client.56663 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi016-74285-62"}]: dispatch 2024-03-26T04:24:22.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:22 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T04:24:22.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:22 smithi016 ceph-mon[29101]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:24:22.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:22 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:24:22.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:22 smithi016 ceph-mon[37215]: pgmap v464: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.5 KiB/s wr, 5 op/s 2024-03-26T04:24:22.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:22 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-26T04:24:22.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:22 smithi016 ceph-mon[37215]: osdmap e337: 8 total, 8 up, 8 in 2024-03-26T04:24:22.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:22 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3721104634' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi016-74285-62"}]: dispatch 2024-03-26T04:24:22.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:22 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T04:24:22.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:22 smithi016 ceph-mon[37215]: from='client.56663 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi016-74285-62"}]: dispatch 2024-03-26T04:24:22.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:22 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T04:24:22.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:22 smithi016 ceph-mon[37215]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:24:22.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:22 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:24:23.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:23 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:24:23.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:23 smithi016 ceph-mon[37215]: from='client.56663 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi016-74285-62"}]': finished 2024-03-26T04:24:23.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:23 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-26T04:24:23.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:23 smithi016 ceph-mon[37215]: osdmap e338: 8 total, 8 up, 8 in 2024-03-26T04:24:23.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:23 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3721104634' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi016-74285-62"}]: dispatch 2024-03-26T04:24:23.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:23 smithi016 ceph-mon[37215]: from='client.56663 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi016-74285-62"}]: dispatch 2024-03-26T04:24:23.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:23 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:24:23.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:23 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:24:23.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:23 smithi016 ceph-mon[37215]: from='client.56663 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi016-74285-62"}]': finished 2024-03-26T04:24:23.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:23 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:24:23.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:23 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:24:23.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:23 smithi016 ceph-mon[29101]: from='client.56663 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi016-74285-62"}]': finished 2024-03-26T04:24:23.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:23 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-26T04:24:23.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:23 smithi016 ceph-mon[29101]: osdmap e338: 8 total, 8 up, 8 in 2024-03-26T04:24:23.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:23 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3721104634' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi016-74285-62"}]: dispatch 2024-03-26T04:24:23.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:23 smithi016 ceph-mon[29101]: from='client.56663 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi016-74285-62"}]: dispatch 2024-03-26T04:24:23.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:23 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:24:23.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:23 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:24:23.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:23 smithi016 ceph-mon[29101]: from='client.56663 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi016-74285-62"}]': finished 2024-03-26T04:24:23.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:23 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:24:23.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:23 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:24:23.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:23 smithi183 ceph-mon[33745]: from='client.56663 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleStatPPNS_smithi016-74285-62"}]': finished 2024-03-26T04:24:23.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:23 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-47","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-26T04:24:23.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:23 smithi183 ceph-mon[33745]: osdmap e338: 8 total, 8 up, 8 in 2024-03-26T04:24:23.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:23 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3721104634' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi016-74285-62"}]: dispatch 2024-03-26T04:24:23.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:23 smithi183 ceph-mon[33745]: from='client.56663 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi016-74285-62"}]: dispatch 2024-03-26T04:24:23.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:23 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:24:23.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:23 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:24:23.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:23 smithi183 ceph-mon[33745]: from='client.56663 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleStatPPNS_smithi016-74285-62"}]': finished 2024-03-26T04:24:23.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:23 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:24:24.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:24 smithi016 ceph-mon[37215]: pgmap v467: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:24:24.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:24 smithi016 ceph-mon[37215]: osdmap e339: 8 total, 8 up, 8 in 2024-03-26T04:24:24.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:24 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-47"}]: dispatch 2024-03-26T04:24:24.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:24 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-47"}]: dispatch 2024-03-26T04:24:24.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:24 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1060374578' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi016-74285-63"}]: dispatch 2024-03-26T04:24:24.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:24 smithi016 ceph-mon[37215]: from='client.56669 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi016-74285-63"}]: dispatch 2024-03-26T04:24:24.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:24 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1060374578' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi016-74285-63"}]: dispatch 2024-03-26T04:24:24.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:24 smithi016 ceph-mon[37215]: from='client.56669 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi016-74285-63"}]: dispatch 2024-03-26T04:24:24.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:24 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1060374578' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi016-74285-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:24:24.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:24 smithi016 ceph-mon[37215]: from='client.56669 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi016-74285-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:24:24.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:24 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-47"}]': finished 2024-03-26T04:24:24.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:24 smithi016 ceph-mon[29101]: pgmap v467: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:24:24.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:24 smithi016 ceph-mon[29101]: osdmap e339: 8 total, 8 up, 8 in 2024-03-26T04:24:24.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:24 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-47"}]: dispatch 2024-03-26T04:24:24.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:24 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-47"}]: dispatch 2024-03-26T04:24:24.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:24 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1060374578' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi016-74285-63"}]: dispatch 2024-03-26T04:24:24.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:24 smithi016 ceph-mon[29101]: from='client.56669 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi016-74285-63"}]: dispatch 2024-03-26T04:24:24.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:24 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1060374578' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi016-74285-63"}]: dispatch 2024-03-26T04:24:24.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:24 smithi016 ceph-mon[29101]: from='client.56669 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi016-74285-63"}]: dispatch 2024-03-26T04:24:24.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:24 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1060374578' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi016-74285-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:24:24.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:24 smithi016 ceph-mon[29101]: from='client.56669 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi016-74285-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:24:24.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:24 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-47"}]': finished 2024-03-26T04:24:24.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:24 smithi183 ceph-mon[33745]: pgmap v467: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:24:24.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:24 smithi183 ceph-mon[33745]: osdmap e339: 8 total, 8 up, 8 in 2024-03-26T04:24:24.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:24 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-47"}]: dispatch 2024-03-26T04:24:24.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:24 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-47"}]: dispatch 2024-03-26T04:24:24.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:24 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1060374578' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi016-74285-63"}]: dispatch 2024-03-26T04:24:24.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:24 smithi183 ceph-mon[33745]: from='client.56669 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi016-74285-63"}]: dispatch 2024-03-26T04:24:24.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:24 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1060374578' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi016-74285-63"}]: dispatch 2024-03-26T04:24:24.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:24 smithi183 ceph-mon[33745]: from='client.56669 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi016-74285-63"}]: dispatch 2024-03-26T04:24:24.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:24 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1060374578' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi016-74285-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:24:24.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:24 smithi183 ceph-mon[33745]: from='client.56669 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi016-74285-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:24:24.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:24 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-47"}]': finished 2024-03-26T04:24:26.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:25 smithi016 ceph-mon[37215]: from='client.56669 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi016-74285-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:24:26.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:25 smithi016 ceph-mon[37215]: osdmap e340: 8 total, 8 up, 8 in 2024-03-26T04:24:26.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:25 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1060374578' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi016-74285-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi016-74285-63"}]: dispatch 2024-03-26T04:24:26.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:25 smithi016 ceph-mon[37215]: from='client.56669 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi016-74285-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi016-74285-63"}]: dispatch 2024-03-26T04:24:26.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:25 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:24:26.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:25 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:24:26.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:25 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-47"}]: dispatch 2024-03-26T04:24:26.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:25 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-47"}]: dispatch 2024-03-26T04:24:26.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:25 smithi016 ceph-mon[29101]: from='client.56669 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi016-74285-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:24:26.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:25 smithi016 ceph-mon[29101]: osdmap e340: 8 total, 8 up, 8 in 2024-03-26T04:24:26.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:25 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1060374578' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi016-74285-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi016-74285-63"}]: dispatch 2024-03-26T04:24:26.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:25 smithi016 ceph-mon[29101]: from='client.56669 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi016-74285-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi016-74285-63"}]: dispatch 2024-03-26T04:24:26.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:25 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:24:26.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:25 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:24:26.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:25 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-47"}]: dispatch 2024-03-26T04:24:26.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:25 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-47"}]: dispatch 2024-03-26T04:24:26.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:25 smithi183 ceph-mon[33745]: from='client.56669 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-StatRemovePP_smithi016-74285-63", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:24:26.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:25 smithi183 ceph-mon[33745]: osdmap e340: 8 total, 8 up, 8 in 2024-03-26T04:24:26.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:25 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1060374578' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi016-74285-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi016-74285-63"}]: dispatch 2024-03-26T04:24:26.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:25 smithi183 ceph-mon[33745]: from='client.56669 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi016-74285-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi016-74285-63"}]: dispatch 2024-03-26T04:24:26.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:25 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:24:26.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:25 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:24:26.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:25 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-47"}]: dispatch 2024-03-26T04:24:26.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:25 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-47"}]: dispatch 2024-03-26T04:24:27.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:26 smithi016 ceph-mon[29101]: pgmap v470: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:24:27.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:26 smithi016 ceph-mon[29101]: osdmap e341: 8 total, 8 up, 8 in 2024-03-26T04:24:27.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:26 smithi016 ceph-mon[29101]: from='client.56669 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi016-74285-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi016-74285-63"}]': finished 2024-03-26T04:24:27.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:26 smithi016 ceph-mon[29101]: osdmap e342: 8 total, 8 up, 8 in 2024-03-26T04:24:27.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:26 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:24:27.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:26 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:24:27.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:26 smithi016 ceph-mon[37215]: pgmap v470: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:24:27.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:26 smithi016 ceph-mon[37215]: osdmap e341: 8 total, 8 up, 8 in 2024-03-26T04:24:27.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:26 smithi016 ceph-mon[37215]: from='client.56669 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi016-74285-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi016-74285-63"}]': finished 2024-03-26T04:24:27.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:26 smithi016 ceph-mon[37215]: osdmap e342: 8 total, 8 up, 8 in 2024-03-26T04:24:27.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:26 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:24:27.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:26 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:24:27.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:26 smithi183 ceph-mon[33745]: pgmap v470: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:24:27.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:26 smithi183 ceph-mon[33745]: osdmap e341: 8 total, 8 up, 8 in 2024-03-26T04:24:27.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:26 smithi183 ceph-mon[33745]: from='client.56669 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "StatRemovePP_smithi016-74285-63", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-StatRemovePP_smithi016-74285-63"}]': finished 2024-03-26T04:24:27.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:26 smithi183 ceph-mon[33745]: osdmap e342: 8 total, 8 up, 8 in 2024-03-26T04:24:27.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:26 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:24:27.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:26 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-49","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:24:28.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:28 smithi016 ceph-mon[37215]: pgmap v473: 300 pgs: 17 creating+peering, 23 unknown, 260 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T04:24:28.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:28 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:24:28.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:28 smithi016 ceph-mon[37215]: osdmap e343: 8 total, 8 up, 8 in 2024-03-26T04:24:28.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:28 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:24:28.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:28 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:24:28.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:28 smithi016 ceph-mon[37215]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:24:28.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:28 smithi016 ceph-mon[29101]: pgmap v473: 300 pgs: 17 creating+peering, 23 unknown, 260 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T04:24:28.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:28 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:24:28.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:28 smithi016 ceph-mon[29101]: osdmap e343: 8 total, 8 up, 8 in 2024-03-26T04:24:28.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:28 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:24:28.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:28 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:24:28.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:28 smithi016 ceph-mon[29101]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:24:28.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:28 smithi183 ceph-mon[33745]: pgmap v473: 300 pgs: 17 creating+peering, 23 unknown, 260 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T04:24:28.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:28 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-49","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:24:28.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:28 smithi183 ceph-mon[33745]: osdmap e343: 8 total, 8 up, 8 in 2024-03-26T04:24:28.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:28 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:24:28.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:28 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-49", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:24:28.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:28 smithi183 ceph-mon[33745]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:24:29.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:24:28 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:24:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:24:29.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:29 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-49", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:24:29.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:29 smithi016 ceph-mon[37215]: osdmap e344: 8 total, 8 up, 8 in 2024-03-26T04:24:29.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:29 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-49"}]: dispatch 2024-03-26T04:24:29.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:29 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1060374578' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi016-74285-63"}]: dispatch 2024-03-26T04:24:29.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:29 smithi016 ceph-mon[37215]: from='client.56669 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi016-74285-63"}]: dispatch 2024-03-26T04:24:29.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:29 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-49"}]: dispatch 2024-03-26T04:24:29.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:29 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-49", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:24:29.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:29 smithi016 ceph-mon[29101]: osdmap e344: 8 total, 8 up, 8 in 2024-03-26T04:24:29.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:29 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-49"}]: dispatch 2024-03-26T04:24:29.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:29 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1060374578' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi016-74285-63"}]: dispatch 2024-03-26T04:24:29.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:29 smithi016 ceph-mon[29101]: from='client.56669 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi016-74285-63"}]: dispatch 2024-03-26T04:24:29.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:29 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-49"}]: dispatch 2024-03-26T04:24:29.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:29 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-49", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:24:29.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:29 smithi183 ceph-mon[33745]: osdmap e344: 8 total, 8 up, 8 in 2024-03-26T04:24:29.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:29 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-49"}]: dispatch 2024-03-26T04:24:29.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:29 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1060374578' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi016-74285-63"}]: dispatch 2024-03-26T04:24:29.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:29 smithi183 ceph-mon[33745]: from='client.56669 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi016-74285-63"}]: dispatch 2024-03-26T04:24:29.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:29 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-49"}]: dispatch 2024-03-26T04:24:30.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:30 smithi016 ceph-mon[29101]: pgmap v476: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T04:24:30.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:30 smithi016 ceph-mon[29101]: from='client.56669 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi016-74285-63"}]': finished 2024-03-26T04:24:30.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:30 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-49"}]': finished 2024-03-26T04:24:30.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:30 smithi016 ceph-mon[29101]: osdmap e345: 8 total, 8 up, 8 in 2024-03-26T04:24:30.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:30 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1060374578' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi016-74285-63"}]: dispatch 2024-03-26T04:24:30.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:30 smithi016 ceph-mon[29101]: from='client.56669 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi016-74285-63"}]: dispatch 2024-03-26T04:24:30.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:30 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-49", "mode": "readproxy"}]: dispatch 2024-03-26T04:24:30.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:30 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-49", "mode": "readproxy"}]: dispatch 2024-03-26T04:24:30.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:30 smithi016 ceph-mon[37215]: pgmap v476: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T04:24:30.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:30 smithi016 ceph-mon[37215]: from='client.56669 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi016-74285-63"}]': finished 2024-03-26T04:24:30.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:30 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-49"}]': finished 2024-03-26T04:24:30.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:30 smithi016 ceph-mon[37215]: osdmap e345: 8 total, 8 up, 8 in 2024-03-26T04:24:30.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:30 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1060374578' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi016-74285-63"}]: dispatch 2024-03-26T04:24:30.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:30 smithi016 ceph-mon[37215]: from='client.56669 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi016-74285-63"}]: dispatch 2024-03-26T04:24:30.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:30 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-49", "mode": "readproxy"}]: dispatch 2024-03-26T04:24:30.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:30 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-49", "mode": "readproxy"}]: dispatch 2024-03-26T04:24:30.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:30 smithi183 ceph-mon[33745]: pgmap v476: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T04:24:30.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:30 smithi183 ceph-mon[33745]: from='client.56669 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-StatRemovePP_smithi016-74285-63"}]': finished 2024-03-26T04:24:30.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:30 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-49"}]': finished 2024-03-26T04:24:30.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:30 smithi183 ceph-mon[33745]: osdmap e345: 8 total, 8 up, 8 in 2024-03-26T04:24:30.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:30 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1060374578' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi016-74285-63"}]: dispatch 2024-03-26T04:24:30.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:30 smithi183 ceph-mon[33745]: from='client.56669 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi016-74285-63"}]: dispatch 2024-03-26T04:24:30.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:30 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-49", "mode": "readproxy"}]: dispatch 2024-03-26T04:24:30.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:30 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-49", "mode": "readproxy"}]: dispatch 2024-03-26T04:24:31.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:31 smithi016 ceph-mon[29101]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:24:31.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:31 smithi016 ceph-mon[29101]: from='client.56669 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi016-74285-63"}]': finished 2024-03-26T04:24:31.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:31 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-49", "mode": "readproxy"}]': finished 2024-03-26T04:24:31.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:31 smithi016 ceph-mon[29101]: osdmap e346: 8 total, 8 up, 8 in 2024-03-26T04:24:31.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:31 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/781584841' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi016-74285-64"}]: dispatch 2024-03-26T04:24:31.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:31 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/781584841' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi016-74285-64"}]: dispatch 2024-03-26T04:24:31.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:31 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/781584841' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi016-74285-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:24:31.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:31 smithi016 ceph-mon[37215]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:24:31.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:31 smithi016 ceph-mon[37215]: from='client.56669 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi016-74285-63"}]': finished 2024-03-26T04:24:31.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:31 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-49", "mode": "readproxy"}]': finished 2024-03-26T04:24:31.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:31 smithi016 ceph-mon[37215]: osdmap e346: 8 total, 8 up, 8 in 2024-03-26T04:24:31.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:31 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/781584841' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi016-74285-64"}]: dispatch 2024-03-26T04:24:31.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:31 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/781584841' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi016-74285-64"}]: dispatch 2024-03-26T04:24:31.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:31 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/781584841' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi016-74285-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:24:31.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:31 smithi183 ceph-mon[33745]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:24:31.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:31 smithi183 ceph-mon[33745]: from='client.56669 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"StatRemovePP_smithi016-74285-63"}]': finished 2024-03-26T04:24:31.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:31 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-49", "mode": "readproxy"}]': finished 2024-03-26T04:24:31.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:31 smithi183 ceph-mon[33745]: osdmap e346: 8 total, 8 up, 8 in 2024-03-26T04:24:31.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:31 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/781584841' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi016-74285-64"}]: dispatch 2024-03-26T04:24:31.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:31 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/781584841' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi016-74285-64"}]: dispatch 2024-03-26T04:24:31.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:31 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/781584841' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi016-74285-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:24:32.638 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:24:32 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:24:32.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:32 smithi016 ceph-mon[37215]: pgmap v479: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:24:32.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:32 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/781584841' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi016-74285-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:24:32.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:32 smithi016 ceph-mon[37215]: osdmap e347: 8 total, 8 up, 8 in 2024-03-26T04:24:32.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:32 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/781584841' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi016-74285-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi016-74285-64"}]: dispatch 2024-03-26T04:24:32.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:32 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:24:32.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:32 smithi016 ceph-mon[29101]: pgmap v479: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:24:32.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:32 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/781584841' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi016-74285-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:24:32.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:32 smithi016 ceph-mon[29101]: osdmap e347: 8 total, 8 up, 8 in 2024-03-26T04:24:32.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:32 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/781584841' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi016-74285-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi016-74285-64"}]: dispatch 2024-03-26T04:24:32.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:32 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:24:33.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:32 smithi183 ceph-mon[33745]: pgmap v479: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 635 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:24:33.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:32 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/781584841' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ExecuteClassPP_smithi016-74285-64", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:24:33.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:32 smithi183 ceph-mon[33745]: osdmap e347: 8 total, 8 up, 8 in 2024-03-26T04:24:33.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:32 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/781584841' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi016-74285-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi016-74285-64"}]: dispatch 2024-03-26T04:24:33.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:32 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:24:33.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:33 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:24:33.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:33 smithi016 ceph-mon[37215]: osdmap e348: 8 total, 8 up, 8 in 2024-03-26T04:24:33.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:33 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:24:33.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:33 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:24:33.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:33 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:24:33.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:33 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/781584841' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi016-74285-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi016-74285-64"}]': finished 2024-03-26T04:24:33.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:33 smithi016 ceph-mon[37215]: osdmap e349: 8 total, 8 up, 8 in 2024-03-26T04:24:33.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:33 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:24:33.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:33 smithi016 ceph-mon[29101]: osdmap e348: 8 total, 8 up, 8 in 2024-03-26T04:24:33.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:33 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:24:33.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:33 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:24:33.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:33 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:24:33.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:33 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/781584841' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi016-74285-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi016-74285-64"}]': finished 2024-03-26T04:24:33.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:33 smithi016 ceph-mon[29101]: osdmap e349: 8 total, 8 up, 8 in 2024-03-26T04:24:34.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:33 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:24:34.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:33 smithi183 ceph-mon[33745]: osdmap e348: 8 total, 8 up, 8 in 2024-03-26T04:24:34.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:33 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:24:34.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:33 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:24:34.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:33 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:24:34.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:33 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/781584841' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ExecuteClassPP_smithi016-74285-64", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ExecuteClassPP_smithi016-74285-64"}]': finished 2024-03-26T04:24:34.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:33 smithi183 ceph-mon[33745]: osdmap e349: 8 total, 8 up, 8 in 2024-03-26T04:24:34.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:34 smithi016 ceph-mon[37215]: pgmap v482: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:24:34.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:34 smithi016 ceph-mon[37215]: osdmap e350: 8 total, 8 up, 8 in 2024-03-26T04:24:34.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:34 smithi016 ceph-mon[29101]: pgmap v482: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:24:34.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:34 smithi016 ceph-mon[29101]: osdmap e350: 8 total, 8 up, 8 in 2024-03-26T04:24:35.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:34 smithi183 ceph-mon[33745]: pgmap v482: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:24:35.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:34 smithi183 ceph-mon[33745]: osdmap e350: 8 total, 8 up, 8 in 2024-03-26T04:24:36.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:36 smithi016 ceph-mon[29101]: pgmap v485: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:24:36.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:36 smithi016 ceph-mon[29101]: osdmap e351: 8 total, 8 up, 8 in 2024-03-26T04:24:36.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:36 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/781584841' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi016-74285-64"}]: dispatch 2024-03-26T04:24:36.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:36 smithi016 ceph-mon[29101]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:24:36.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:36 smithi016 ceph-mon[37215]: pgmap v485: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:24:36.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:36 smithi016 ceph-mon[37215]: osdmap e351: 8 total, 8 up, 8 in 2024-03-26T04:24:36.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:36 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/781584841' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi016-74285-64"}]: dispatch 2024-03-26T04:24:36.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:36 smithi016 ceph-mon[37215]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:24:36.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:36 smithi183 ceph-mon[33745]: pgmap v485: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:24:36.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:36 smithi183 ceph-mon[33745]: osdmap e351: 8 total, 8 up, 8 in 2024-03-26T04:24:36.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:36 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/781584841' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi016-74285-64"}]: dispatch 2024-03-26T04:24:36.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:36 smithi183 ceph-mon[33745]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:24:37.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:37 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/781584841' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi016-74285-64"}]': finished 2024-03-26T04:24:37.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:37 smithi016 ceph-mon[37215]: osdmap e352: 8 total, 8 up, 8 in 2024-03-26T04:24:37.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:37 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/781584841' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi016-74285-64"}]: dispatch 2024-03-26T04:24:37.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:37 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:24:37.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:37 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:24:37.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:37 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/781584841' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi016-74285-64"}]': finished 2024-03-26T04:24:37.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:37 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/781584841' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi016-74285-64"}]': finished 2024-03-26T04:24:37.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:37 smithi016 ceph-mon[29101]: osdmap e352: 8 total, 8 up, 8 in 2024-03-26T04:24:37.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:37 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/781584841' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi016-74285-64"}]: dispatch 2024-03-26T04:24:37.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:37 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:24:37.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:37 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:24:37.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:37 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/781584841' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi016-74285-64"}]': finished 2024-03-26T04:24:37.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:37 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/781584841' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ExecuteClassPP_smithi016-74285-64"}]': finished 2024-03-26T04:24:37.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:37 smithi183 ceph-mon[33745]: osdmap e352: 8 total, 8 up, 8 in 2024-03-26T04:24:37.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:37 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/781584841' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi016-74285-64"}]: dispatch 2024-03-26T04:24:37.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:37 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:24:37.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:37 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:24:37.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:37 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/781584841' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ExecuteClassPP_smithi016-74285-64"}]': finished 2024-03-26T04:24:38.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:38 smithi016 ceph-mon[37215]: pgmap v488: 292 pgs: 292 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:24:38.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:38 smithi016 ceph-mon[37215]: osdmap e353: 8 total, 8 up, 8 in 2024-03-26T04:24:38.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:38 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/642131036' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi016-74285-65"}]: dispatch 2024-03-26T04:24:38.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:38 smithi016 ceph-mon[37215]: from='client.56681 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi016-74285-65"}]: dispatch 2024-03-26T04:24:38.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:38 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/642131036' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi016-74285-65"}]: dispatch 2024-03-26T04:24:38.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:38 smithi016 ceph-mon[37215]: from='client.56681 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi016-74285-65"}]: dispatch 2024-03-26T04:24:38.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:38 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/642131036' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi016-74285-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:24:38.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:38 smithi016 ceph-mon[37215]: from='client.56681 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi016-74285-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:24:38.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:38 smithi016 ceph-mon[29101]: pgmap v488: 292 pgs: 292 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:24:38.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:38 smithi016 ceph-mon[29101]: osdmap e353: 8 total, 8 up, 8 in 2024-03-26T04:24:38.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:38 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/642131036' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi016-74285-65"}]: dispatch 2024-03-26T04:24:38.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:38 smithi016 ceph-mon[29101]: from='client.56681 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi016-74285-65"}]: dispatch 2024-03-26T04:24:38.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:38 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/642131036' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi016-74285-65"}]: dispatch 2024-03-26T04:24:38.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:38 smithi016 ceph-mon[29101]: from='client.56681 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi016-74285-65"}]: dispatch 2024-03-26T04:24:38.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:38 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/642131036' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi016-74285-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:24:38.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:38 smithi016 ceph-mon[29101]: from='client.56681 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi016-74285-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:24:38.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:38 smithi183 ceph-mon[33745]: pgmap v488: 292 pgs: 292 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:24:38.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:38 smithi183 ceph-mon[33745]: osdmap e353: 8 total, 8 up, 8 in 2024-03-26T04:24:38.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:38 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/642131036' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi016-74285-65"}]: dispatch 2024-03-26T04:24:38.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:38 smithi183 ceph-mon[33745]: from='client.56681 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi016-74285-65"}]: dispatch 2024-03-26T04:24:38.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:38 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/642131036' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi016-74285-65"}]: dispatch 2024-03-26T04:24:38.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:38 smithi183 ceph-mon[33745]: from='client.56681 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi016-74285-65"}]: dispatch 2024-03-26T04:24:38.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:38 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/642131036' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi016-74285-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:24:38.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:38 smithi183 ceph-mon[33745]: from='client.56681 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi016-74285-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:24:39.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:24:38 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:24:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:24:39.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:39 smithi016 ceph-mon[29101]: from='client.56681 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi016-74285-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:24:39.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:39 smithi016 ceph-mon[29101]: osdmap e354: 8 total, 8 up, 8 in 2024-03-26T04:24:39.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:39 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/642131036' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi016-74285-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi016-74285-65"}]: dispatch 2024-03-26T04:24:39.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:39 smithi016 ceph-mon[29101]: from='client.56681 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi016-74285-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi016-74285-65"}]: dispatch 2024-03-26T04:24:39.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:39 smithi016 ceph-mon[37215]: from='client.56681 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi016-74285-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:24:39.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:39 smithi016 ceph-mon[37215]: osdmap e354: 8 total, 8 up, 8 in 2024-03-26T04:24:39.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:39 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/642131036' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi016-74285-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi016-74285-65"}]: dispatch 2024-03-26T04:24:39.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:39 smithi016 ceph-mon[37215]: from='client.56681 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi016-74285-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi016-74285-65"}]: dispatch 2024-03-26T04:24:39.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:39 smithi183 ceph-mon[33745]: from='client.56681 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-OmapPP_smithi016-74285-65", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:24:39.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:39 smithi183 ceph-mon[33745]: osdmap e354: 8 total, 8 up, 8 in 2024-03-26T04:24:39.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:39 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/642131036' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi016-74285-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi016-74285-65"}]: dispatch 2024-03-26T04:24:39.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:39 smithi183 ceph-mon[33745]: from='client.56681 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "OmapPP_smithi016-74285-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi016-74285-65"}]: dispatch 2024-03-26T04:24:40.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:40 smithi016 ceph-mon[37215]: pgmap v491: 292 pgs: 292 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:24:40.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:40 smithi016 ceph-mon[37215]: osdmap e355: 8 total, 8 up, 8 in 2024-03-26T04:24:40.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:40 smithi016 ceph-mon[29101]: pgmap v491: 292 pgs: 292 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:24:40.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:40 smithi016 ceph-mon[29101]: osdmap e355: 8 total, 8 up, 8 in 2024-03-26T04:24:40.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:40 smithi183 ceph-mon[33745]: pgmap v491: 292 pgs: 292 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:24:40.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:40 smithi183 ceph-mon[33745]: osdmap e355: 8 total, 8 up, 8 in 2024-03-26T04:24:41.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:41 smithi183 ceph-mon[33745]: from='client.56681 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi016-74285-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi016-74285-65"}]': finished 2024-03-26T04:24:41.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:41 smithi183 ceph-mon[33745]: osdmap e356: 8 total, 8 up, 8 in 2024-03-26T04:24:41.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:41 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:24:41.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:41 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:24:41.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:41 smithi183 ceph-mon[33745]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:24:41.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:41 smithi016 ceph-mon[29101]: from='client.56681 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi016-74285-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi016-74285-65"}]': finished 2024-03-26T04:24:41.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:41 smithi016 ceph-mon[29101]: osdmap e356: 8 total, 8 up, 8 in 2024-03-26T04:24:41.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:41 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:24:41.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:41 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:24:41.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:41 smithi016 ceph-mon[29101]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:24:41.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:41 smithi016 ceph-mon[37215]: from='client.56681 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "OmapPP_smithi016-74285-65", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-OmapPP_smithi016-74285-65"}]': finished 2024-03-26T04:24:41.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:41 smithi016 ceph-mon[37215]: osdmap e356: 8 total, 8 up, 8 in 2024-03-26T04:24:41.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:41 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:24:41.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:41 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:24:41.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:41 smithi016 ceph-mon[37215]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:24:42.792 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:24:42 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:24:42.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:42 smithi183 ceph-mon[33745]: pgmap v494: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:24:42.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:42 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:24:42.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:42 smithi183 ceph-mon[33745]: osdmap e357: 8 total, 8 up, 8 in 2024-03-26T04:24:42.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:42 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-49"}]: dispatch 2024-03-26T04:24:42.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:42 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-49"}]: dispatch 2024-03-26T04:24:42.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:42 smithi183 ceph-mon[33745]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:24:42.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:42 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-49"}]': finished 2024-03-26T04:24:42.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:42 smithi183 ceph-mon[33745]: osdmap e358: 8 total, 8 up, 8 in 2024-03-26T04:24:42.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:42 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/642131036' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi016-74285-65"}]: dispatch 2024-03-26T04:24:42.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:42 smithi183 ceph-mon[33745]: from='client.56681 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi016-74285-65"}]: dispatch 2024-03-26T04:24:42.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:42 smithi016 ceph-mon[37215]: pgmap v494: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:24:42.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:42 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:24:42.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:42 smithi016 ceph-mon[37215]: osdmap e357: 8 total, 8 up, 8 in 2024-03-26T04:24:42.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:42 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-49"}]: dispatch 2024-03-26T04:24:42.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:42 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-49"}]: dispatch 2024-03-26T04:24:42.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:42 smithi016 ceph-mon[37215]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:24:42.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:42 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-49"}]': finished 2024-03-26T04:24:42.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:42 smithi016 ceph-mon[37215]: osdmap e358: 8 total, 8 up, 8 in 2024-03-26T04:24:42.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:42 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/642131036' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi016-74285-65"}]: dispatch 2024-03-26T04:24:42.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:42 smithi016 ceph-mon[37215]: from='client.56681 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi016-74285-65"}]: dispatch 2024-03-26T04:24:42.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:42 smithi016 ceph-mon[29101]: pgmap v494: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 640 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:24:42.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:42 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:24:42.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:42 smithi016 ceph-mon[29101]: osdmap e357: 8 total, 8 up, 8 in 2024-03-26T04:24:42.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:42 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-49"}]: dispatch 2024-03-26T04:24:42.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:42 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-49"}]: dispatch 2024-03-26T04:24:42.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:42 smithi016 ceph-mon[29101]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:24:42.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:42 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-49"}]': finished 2024-03-26T04:24:42.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:42 smithi016 ceph-mon[29101]: osdmap e358: 8 total, 8 up, 8 in 2024-03-26T04:24:42.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:42 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/642131036' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi016-74285-65"}]: dispatch 2024-03-26T04:24:42.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:42 smithi016 ceph-mon[29101]: from='client.56681 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi016-74285-65"}]: dispatch 2024-03-26T04:24:43.425 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2:163:head 2024-03-26T04:24:43.425 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:5d165639:::164:head 2024-03-26T04:24:43.425 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:f43765fc:::165:head 2024-03-26T04:24:43.425 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:b4c720e9:::166:head 2024-03-26T04:24:43.425 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:e694b040:::167:head 2024-03-26T04:24:43.425 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:afa38db2:::168:head 2024-03-26T04:24:43.425 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:77ba9f53:::169:head 2024-03-26T04:24:43.425 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:87495034:::170:head 2024-03-26T04:24:43.425 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:7c96bf0e:::171:head 2024-03-26T04:24:43.425 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:dbe346cc:::172:head 2024-03-26T04:24:43.426 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:e943ec24:::173:head 2024-03-26T04:24:43.426 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:f97a9c0c:::174:head 2024-03-26T04:24:43.426 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:6f26e74d:::175:head 2024-03-26T04:24:43.426 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:4f95e106:::176:head 2024-03-26T04:24:43.426 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:0e6f2f8f:::177:head 2024-03-26T04:24:43.426 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:05db05f1:::178:head 2024-03-26T04:24:43.426 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:38a78d66:::179:head 2024-03-26T04:24:43.426 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:d095610b:::180:head 2024-03-26T04:24:43.426 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:a1a9d709:::181:head 2024-03-26T04:24:43.426 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:1e5d39db:::182:head 2024-03-26T04:24:43.426 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:f7df4fb9:::183:head 2024-03-26T04:24:43.426 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:03a7f161:::184:head 2024-03-26T04:24:43.426 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:ba70721e:::185:head 2024-03-26T04:24:43.426 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:28e5662d:::186:head 2024-03-26T04:24:43.427 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:973d52de:::187:head 2024-03-26T04:24:43.427 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:4303eb1c:::188:head 2024-03-26T04:24:43.427 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:b990b48e:::189:head 2024-03-26T04:24:43.427 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:29b8165b:::190:head 2024-03-26T04:24:43.427 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:3547f197:::191:head 2024-03-26T04:24:43.427 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:7e260936:::192:head 2024-03-26T04:24:43.427 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:1abec7b1:::193:head 2024-03-26T04:24:43.427 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:10fdda93:::194:head 2024-03-26T04:24:43.427 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:15817eea:::195:head 2024-03-26T04:24:43.427 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:770bab57:::196:head 2024-03-26T04:24:43.427 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:ed9e13e7:::197:head 2024-03-26T04:24:43.427 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:71471a8f:::198:head 2024-03-26T04:24:43.427 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: checking for 262:10fb1d02:::199:head 2024-03-26T04:24:43.427 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (8039 ms) 2024-03-26T04:24:43.427 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-03-26T04:24:43.428 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: got ls 1711427039,0 2024-03-26T04:24:43.428 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: first is 1711427039 2024-03-26T04:24:43.428 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: got ls 1711427039,0 2024-03-26T04:24:43.428 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: got ls 1711427039,0 2024-03-26T04:24:43.428 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: got ls 1711427039,1711427041,1711427042,0 2024-03-26T04:24:43.428 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: got ls 1711427039,1711427041,1711427042,0 2024-03-26T04:24:43.428 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: got ls 1711427039,1711427041,1711427042,0 2024-03-26T04:24:43.428 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: got ls 1711427039,1711427041,1711427042,1711427044,1711427045,0 2024-03-26T04:24:43.428 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: got ls 1711427039,1711427041,1711427042,1711427044,1711427045,0 2024-03-26T04:24:43.428 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: got ls 1711427039,1711427041,1711427042,1711427044,1711427045,0 2024-03-26T04:24:43.428 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: got ls 1711427039,1711427041,1711427042,1711427044,1711427045,1711427047,1711427048,0 2024-03-26T04:24:43.428 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: got ls 1711427039,1711427041,1711427042,1711427044,1711427045,1711427047,1711427048,0 2024-03-26T04:24:43.428 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: got ls 1711427039,1711427041,1711427042,1711427044,1711427045,1711427047,1711427048,0 2024-03-26T04:24:43.428 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: got ls 1711427042,1711427044,1711427045,1711427047,1711427048,1711427050,1711427051,0 2024-03-26T04:24:43.428 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: first now 1711427042, trimmed 2024-03-26T04:24:43.429 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (20356 ms) 2024-03-26T04:24:43.429 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-03-26T04:24:43.429 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: foo0 2024-03-26T04:24:43.429 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-03-26T04:24:43.429 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (14042 ms) 2024-03-26T04:24:43.429 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-03-26T04:24:44.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:43 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:24:44.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:43 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:24:44.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:43 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:24:44.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:43 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-49"}]: dispatch 2024-03-26T04:24:44.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:43 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-49"}]: dispatch 2024-03-26T04:24:44.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:43 smithi183 ceph-mon[33745]: from='client.56681 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi016-74285-65"}]': finished 2024-03-26T04:24:44.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:43 smithi183 ceph-mon[33745]: osdmap e359: 8 total, 8 up, 8 in 2024-03-26T04:24:44.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:43 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/642131036' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi016-74285-65"}]: dispatch 2024-03-26T04:24:44.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:43 smithi183 ceph-mon[33745]: from='client.56681 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi016-74285-65"}]: dispatch 2024-03-26T04:24:44.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:43 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:24:44.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:43 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:24:44.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:43 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:24:44.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:43 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-49"}]: dispatch 2024-03-26T04:24:44.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:43 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-49"}]: dispatch 2024-03-26T04:24:44.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:43 smithi016 ceph-mon[37215]: from='client.56681 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi016-74285-65"}]': finished 2024-03-26T04:24:44.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:43 smithi016 ceph-mon[37215]: osdmap e359: 8 total, 8 up, 8 in 2024-03-26T04:24:44.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:43 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/642131036' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi016-74285-65"}]: dispatch 2024-03-26T04:24:44.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:43 smithi016 ceph-mon[37215]: from='client.56681 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi016-74285-65"}]: dispatch 2024-03-26T04:24:44.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:43 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:24:44.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:43 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:24:44.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:43 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:24:44.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:43 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-49"}]: dispatch 2024-03-26T04:24:44.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:43 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-49"}]: dispatch 2024-03-26T04:24:44.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:43 smithi016 ceph-mon[29101]: from='client.56681 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-OmapPP_smithi016-74285-65"}]': finished 2024-03-26T04:24:44.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:43 smithi016 ceph-mon[29101]: osdmap e359: 8 total, 8 up, 8 in 2024-03-26T04:24:44.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:43 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/642131036' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi016-74285-65"}]: dispatch 2024-03-26T04:24:44.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:43 smithi016 ceph-mon[29101]: from='client.56681 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi016-74285-65"}]: dispatch 2024-03-26T04:24:45.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:44 smithi183 ceph-mon[33745]: pgmap v497: 292 pgs: 292 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:24:45.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:44 smithi183 ceph-mon[33745]: from='client.56681 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi016-74285-65"}]': finished 2024-03-26T04:24:45.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:44 smithi183 ceph-mon[33745]: osdmap e360: 8 total, 8 up, 8 in 2024-03-26T04:24:45.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:44 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:24:45.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:44 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:24:45.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:44 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2017806794' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi016-74285-66"}]: dispatch 2024-03-26T04:24:45.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:44 smithi183 ceph-mon[33745]: from='client.56687 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi016-74285-66"}]: dispatch 2024-03-26T04:24:45.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:44 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2017806794' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi016-74285-66"}]: dispatch 2024-03-26T04:24:45.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:44 smithi183 ceph-mon[33745]: from='client.56687 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi016-74285-66"}]: dispatch 2024-03-26T04:24:45.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:44 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2017806794' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi016-74285-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:24:45.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:44 smithi183 ceph-mon[33745]: from='client.56687 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi016-74285-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:24:45.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:44 smithi016 ceph-mon[37215]: pgmap v497: 292 pgs: 292 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:24:45.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:44 smithi016 ceph-mon[37215]: from='client.56681 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi016-74285-65"}]': finished 2024-03-26T04:24:45.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:44 smithi016 ceph-mon[37215]: osdmap e360: 8 total, 8 up, 8 in 2024-03-26T04:24:45.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:44 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:24:45.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:44 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:24:45.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:44 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2017806794' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi016-74285-66"}]: dispatch 2024-03-26T04:24:45.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:44 smithi016 ceph-mon[37215]: from='client.56687 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi016-74285-66"}]: dispatch 2024-03-26T04:24:45.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:44 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2017806794' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi016-74285-66"}]: dispatch 2024-03-26T04:24:45.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:44 smithi016 ceph-mon[37215]: from='client.56687 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi016-74285-66"}]: dispatch 2024-03-26T04:24:45.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:44 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2017806794' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi016-74285-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:24:45.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:44 smithi016 ceph-mon[37215]: from='client.56687 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi016-74285-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:24:45.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:44 smithi016 ceph-mon[29101]: pgmap v497: 292 pgs: 292 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:24:45.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:44 smithi016 ceph-mon[29101]: from='client.56681 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"OmapPP_smithi016-74285-65"}]': finished 2024-03-26T04:24:45.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:44 smithi016 ceph-mon[29101]: osdmap e360: 8 total, 8 up, 8 in 2024-03-26T04:24:45.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:44 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:24:45.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:44 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-51","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:24:45.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:44 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2017806794' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi016-74285-66"}]: dispatch 2024-03-26T04:24:45.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:44 smithi016 ceph-mon[29101]: from='client.56687 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi016-74285-66"}]: dispatch 2024-03-26T04:24:45.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:44 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2017806794' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi016-74285-66"}]: dispatch 2024-03-26T04:24:45.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:44 smithi016 ceph-mon[29101]: from='client.56687 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi016-74285-66"}]: dispatch 2024-03-26T04:24:45.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:44 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2017806794' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi016-74285-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:24:45.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:44 smithi016 ceph-mon[29101]: from='client.56687 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi016-74285-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:24:46.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:46 smithi183 ceph-mon[33745]: pgmap v500: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:24:46.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:46 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:24:46.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:46 smithi183 ceph-mon[33745]: from='client.56687 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi016-74285-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:24:46.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:46 smithi183 ceph-mon[33745]: osdmap e361: 8 total, 8 up, 8 in 2024-03-26T04:24:46.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:46 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2017806794' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi016-74285-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi016-74285-66"}]: dispatch 2024-03-26T04:24:46.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:46 smithi183 ceph-mon[33745]: from='client.56687 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi016-74285-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi016-74285-66"}]: dispatch 2024-03-26T04:24:46.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:46 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:24:46.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:46 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:24:46.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:46 smithi183 ceph-mon[33745]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:24:46.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:46 smithi016 ceph-mon[37215]: pgmap v500: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:24:46.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:46 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:24:46.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:46 smithi016 ceph-mon[37215]: from='client.56687 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi016-74285-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:24:46.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:46 smithi016 ceph-mon[37215]: osdmap e361: 8 total, 8 up, 8 in 2024-03-26T04:24:46.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:46 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2017806794' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi016-74285-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi016-74285-66"}]: dispatch 2024-03-26T04:24:46.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:46 smithi016 ceph-mon[37215]: from='client.56687 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi016-74285-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi016-74285-66"}]: dispatch 2024-03-26T04:24:46.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:46 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:24:46.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:46 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:24:46.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:46 smithi016 ceph-mon[37215]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:24:46.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:46 smithi016 ceph-mon[29101]: pgmap v500: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:24:46.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:46 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-51","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:24:46.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:46 smithi016 ceph-mon[29101]: from='client.56687 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-MultiWritePP_smithi016-74285-66", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:24:46.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:46 smithi016 ceph-mon[29101]: osdmap e361: 8 total, 8 up, 8 in 2024-03-26T04:24:46.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:46 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2017806794' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi016-74285-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi016-74285-66"}]: dispatch 2024-03-26T04:24:46.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:46 smithi016 ceph-mon[29101]: from='client.56687 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi016-74285-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi016-74285-66"}]: dispatch 2024-03-26T04:24:46.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:46 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:24:46.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:46 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-51", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:24:46.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:46 smithi016 ceph-mon[29101]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:24:47.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:47 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-51", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:24:47.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:47 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-51"}]: dispatch 2024-03-26T04:24:47.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:47 smithi183 ceph-mon[33745]: osdmap e362: 8 total, 8 up, 8 in 2024-03-26T04:24:47.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:47 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-51"}]: dispatch 2024-03-26T04:24:47.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:47 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-51", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:24:47.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:47 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-51"}]: dispatch 2024-03-26T04:24:47.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:47 smithi016 ceph-mon[37215]: osdmap e362: 8 total, 8 up, 8 in 2024-03-26T04:24:47.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:47 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-51"}]: dispatch 2024-03-26T04:24:47.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:47 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-51", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:24:47.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:47 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-51"}]: dispatch 2024-03-26T04:24:47.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:47 smithi016 ceph-mon[29101]: osdmap e362: 8 total, 8 up, 8 in 2024-03-26T04:24:47.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:47 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-51"}]: dispatch 2024-03-26T04:24:48.746 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:48 smithi016 ceph-mon[37215]: pgmap v503: 292 pgs: 17 unknown, 275 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:24:48.746 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:48 smithi016 ceph-mon[37215]: from='client.56687 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi016-74285-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi016-74285-66"}]': finished 2024-03-26T04:24:48.746 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:48 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-51"}]': finished 2024-03-26T04:24:48.746 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:48 smithi016 ceph-mon[37215]: osdmap e363: 8 total, 8 up, 8 in 2024-03-26T04:24:48.746 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:48 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-51", "mode": "writeback"}]: dispatch 2024-03-26T04:24:48.746 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:48 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-51", "mode": "writeback"}]: dispatch 2024-03-26T04:24:48.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:48 smithi016 ceph-mon[29101]: pgmap v503: 292 pgs: 17 unknown, 275 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:24:48.747 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:48 smithi016 ceph-mon[29101]: from='client.56687 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi016-74285-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi016-74285-66"}]': finished 2024-03-26T04:24:48.747 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:48 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-51"}]': finished 2024-03-26T04:24:48.747 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:48 smithi016 ceph-mon[29101]: osdmap e363: 8 total, 8 up, 8 in 2024-03-26T04:24:48.747 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:48 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-51", "mode": "writeback"}]: dispatch 2024-03-26T04:24:48.747 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:48 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-51", "mode": "writeback"}]: dispatch 2024-03-26T04:24:48.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:48 smithi183 ceph-mon[33745]: pgmap v503: 292 pgs: 17 unknown, 275 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:24:48.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:48 smithi183 ceph-mon[33745]: from='client.56687 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "MultiWritePP_smithi016-74285-66", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-MultiWritePP_smithi016-74285-66"}]': finished 2024-03-26T04:24:48.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:48 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-51"}]': finished 2024-03-26T04:24:48.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:48 smithi183 ceph-mon[33745]: osdmap e363: 8 total, 8 up, 8 in 2024-03-26T04:24:48.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:48 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-51", "mode": "writeback"}]: dispatch 2024-03-26T04:24:48.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:48 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-51", "mode": "writeback"}]: dispatch 2024-03-26T04:24:49.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:24:48 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:24:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:24:49.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:49 smithi183 ceph-mon[33745]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:24:49.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:49 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-51", "mode": "writeback"}]': finished 2024-03-26T04:24:49.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:49 smithi183 ceph-mon[33745]: osdmap e364: 8 total, 8 up, 8 in 2024-03-26T04:24:49.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:49 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:24:49.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:49 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:24:49.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:49 smithi016 ceph-mon[37215]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:24:49.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:49 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-51", "mode": "writeback"}]': finished 2024-03-26T04:24:49.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:49 smithi016 ceph-mon[37215]: osdmap e364: 8 total, 8 up, 8 in 2024-03-26T04:24:49.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:49 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:24:49.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:49 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:24:49.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:49 smithi016 ceph-mon[29101]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:24:49.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:49 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-51", "mode": "writeback"}]': finished 2024-03-26T04:24:49.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:49 smithi016 ceph-mon[29101]: osdmap e364: 8 total, 8 up, 8 in 2024-03-26T04:24:49.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:49 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:24:49.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:49 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-51","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:24:50.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:50 smithi183 ceph-mon[33745]: pgmap v506: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T04:24:50.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:50 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-51","var": "hit_set_count","val": "2"}]': finished 2024-03-26T04:24:50.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:50 smithi183 ceph-mon[33745]: osdmap e365: 8 total, 8 up, 8 in 2024-03-26T04:24:50.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:50 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2017806794' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi016-74285-66"}]: dispatch 2024-03-26T04:24:50.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:50 smithi183 ceph-mon[33745]: from='client.56687 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi016-74285-66"}]: dispatch 2024-03-26T04:24:50.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:50 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:24:50.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:50 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:24:50.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:50 smithi016 ceph-mon[37215]: pgmap v506: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T04:24:50.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:50 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-51","var": "hit_set_count","val": "2"}]': finished 2024-03-26T04:24:50.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:50 smithi016 ceph-mon[37215]: osdmap e365: 8 total, 8 up, 8 in 2024-03-26T04:24:50.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:50 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2017806794' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi016-74285-66"}]: dispatch 2024-03-26T04:24:50.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:50 smithi016 ceph-mon[37215]: from='client.56687 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi016-74285-66"}]: dispatch 2024-03-26T04:24:50.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:50 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:24:50.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:50 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:24:50.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:50 smithi016 ceph-mon[29101]: pgmap v506: 300 pgs: 8 unknown, 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1023 B/s wr, 2 op/s 2024-03-26T04:24:50.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:50 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-51","var": "hit_set_count","val": "2"}]': finished 2024-03-26T04:24:50.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:50 smithi016 ceph-mon[29101]: osdmap e365: 8 total, 8 up, 8 in 2024-03-26T04:24:50.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:50 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2017806794' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi016-74285-66"}]: dispatch 2024-03-26T04:24:50.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:50 smithi016 ceph-mon[29101]: from='client.56687 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi016-74285-66"}]: dispatch 2024-03-26T04:24:50.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:50 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:24:50.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:50 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-51","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:24:51.474 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ 2024-03-26T04:24:51.474 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3026 ms) 2024-03-26T04:24:51.474 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-03-26T04:24:51.474 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (7037 ms) 2024-03-26T04:24:51.474 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-03-26T04:24:51.474 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (7012 ms) 2024-03-26T04:24:51.474 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-03-26T04:24:51.474 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (7023 ms) 2024-03-26T04:24:51.474 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-03-26T04:24:51.475 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (7023 ms) 2024-03-26T04:24:51.475 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-03-26T04:24:51.475 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (7039 ms) 2024-03-26T04:24:51.475 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-03-26T04:24:51.475 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7036 ms) 2024-03-26T04:24:51.475 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (139320 ms total) 2024-03-26T04:24:51.475 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: 2024-03-26T04:24:51.475 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-03-26T04:24:51.475 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (302938 ms total) 2024-03-26T04:24:51.475 INFO:tasks.workunit.client.0.smithi016.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-03-26T04:24:51.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:51 smithi183 ceph-mon[33745]: from='client.56687 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi016-74285-66"}]': finished 2024-03-26T04:24:51.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:51 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-51","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:24:51.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:51 smithi183 ceph-mon[33745]: osdmap e366: 8 total, 8 up, 8 in 2024-03-26T04:24:51.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:51 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2017806794' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi016-74285-66"}]: dispatch 2024-03-26T04:24:51.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:51 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:24:51.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:51 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:24:51.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:51 smithi183 ceph-mon[33745]: from='client.56687 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi016-74285-66"}]: dispatch 2024-03-26T04:24:51.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:51 smithi016 ceph-mon[37215]: from='client.56687 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi016-74285-66"}]': finished 2024-03-26T04:24:51.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:51 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-51","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:24:51.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:51 smithi016 ceph-mon[37215]: osdmap e366: 8 total, 8 up, 8 in 2024-03-26T04:24:51.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:51 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2017806794' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi016-74285-66"}]: dispatch 2024-03-26T04:24:51.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:51 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:24:51.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:51 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:24:51.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:51 smithi016 ceph-mon[37215]: from='client.56687 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi016-74285-66"}]: dispatch 2024-03-26T04:24:51.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:51 smithi016 ceph-mon[29101]: from='client.56687 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-MultiWritePP_smithi016-74285-66"}]': finished 2024-03-26T04:24:51.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:51 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-51","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:24:51.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:51 smithi016 ceph-mon[29101]: osdmap e366: 8 total, 8 up, 8 in 2024-03-26T04:24:51.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:51 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2017806794' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi016-74285-66"}]: dispatch 2024-03-26T04:24:51.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:51 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:24:51.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:51 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-51","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:24:51.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:51 smithi016 ceph-mon[29101]: from='client.56687 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi016-74285-66"}]: dispatch 2024-03-26T04:24:52.791 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:24:52 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:24:52.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:52 smithi183 ceph-mon[33745]: pgmap v509: 292 pgs: 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 767 B/s wr, 0 op/s 2024-03-26T04:24:52.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:52 smithi183 ceph-mon[33745]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:24:52.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:52 smithi183 ceph-mon[33745]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:24:52.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:52 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-51","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T04:24:52.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:52 smithi183 ceph-mon[33745]: from='client.56687 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi016-74285-66"}]': finished 2024-03-26T04:24:52.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:52 smithi183 ceph-mon[33745]: osdmap e367: 8 total, 8 up, 8 in 2024-03-26T04:24:52.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:52 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T04:24:52.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:52 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T04:24:52.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:52 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:24:52.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:52 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:24:52.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:52 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "275.2", "id": [2, 4]}]: dispatch 2024-03-26T04:24:52.793 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:52 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "275.2", "id": [2, 4]}]: dispatch 2024-03-26T04:24:52.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:52 smithi016 ceph-mon[37215]: pgmap v509: 292 pgs: 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 767 B/s wr, 0 op/s 2024-03-26T04:24:52.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:52 smithi016 ceph-mon[37215]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:24:52.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:52 smithi016 ceph-mon[37215]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:24:52.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:52 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-51","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T04:24:52.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:52 smithi016 ceph-mon[37215]: from='client.56687 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi016-74285-66"}]': finished 2024-03-26T04:24:52.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:52 smithi016 ceph-mon[37215]: osdmap e367: 8 total, 8 up, 8 in 2024-03-26T04:24:52.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:52 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T04:24:52.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:52 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T04:24:52.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:52 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:24:52.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:52 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:24:52.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:52 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "275.2", "id": [2, 4]}]: dispatch 2024-03-26T04:24:52.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:52 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "275.2", "id": [2, 4]}]: dispatch 2024-03-26T04:24:52.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:52 smithi016 ceph-mon[29101]: pgmap v509: 292 pgs: 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 767 B/s wr, 0 op/s 2024-03-26T04:24:52.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:52 smithi016 ceph-mon[29101]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:24:52.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:52 smithi016 ceph-mon[29101]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:24:52.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:52 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-51","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T04:24:52.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:52 smithi016 ceph-mon[29101]: from='client.56687 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"MultiWritePP_smithi016-74285-66"}]': finished 2024-03-26T04:24:52.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:52 smithi016 ceph-mon[29101]: osdmap e367: 8 total, 8 up, 8 in 2024-03-26T04:24:52.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:52 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T04:24:52.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:52 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-51","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T04:24:52.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:52 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:24:52.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:52 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:24:52.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:52 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "275.2", "id": [2, 4]}]: dispatch 2024-03-26T04:24:52.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:52 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "275.2", "id": [2, 4]}]: dispatch 2024-03-26T04:24:53.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:53 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:24:53.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:53 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T04:24:53.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:53 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "275.2", "id": [2, 4]}]': finished 2024-03-26T04:24:53.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:53 smithi183 ceph-mon[33745]: osdmap e368: 8 total, 8 up, 8 in 2024-03-26T04:24:53.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:53 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T04:24:53.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:53 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T04:24:53.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:53 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-51","var": "target_max_objects","val": "1"}]': finished 2024-03-26T04:24:53.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:53 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:24:53.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:53 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T04:24:53.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:53 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "275.2", "id": [2, 4]}]': finished 2024-03-26T04:24:53.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:53 smithi016 ceph-mon[37215]: osdmap e368: 8 total, 8 up, 8 in 2024-03-26T04:24:53.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:53 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T04:24:53.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:53 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T04:24:53.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:53 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-51","var": "target_max_objects","val": "1"}]': finished 2024-03-26T04:24:53.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:53 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:24:53.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:53 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-51","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T04:24:53.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:53 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "275.2", "id": [2, 4]}]': finished 2024-03-26T04:24:53.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:53 smithi016 ceph-mon[29101]: osdmap e368: 8 total, 8 up, 8 in 2024-03-26T04:24:53.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:53 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T04:24:53.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:53 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-51","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T04:24:53.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:53 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-51","var": "target_max_objects","val": "1"}]': finished 2024-03-26T04:24:54.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:54 smithi183 ceph-mon[33745]: pgmap v512: 292 pgs: 292 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-26T04:24:54.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:54 smithi183 ceph-mon[33745]: osdmap e369: 8 total, 8 up, 8 in 2024-03-26T04:24:54.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:54 smithi016 ceph-mon[37215]: pgmap v512: 292 pgs: 292 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-26T04:24:54.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:54 smithi016 ceph-mon[37215]: osdmap e369: 8 total, 8 up, 8 in 2024-03-26T04:24:54.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:54 smithi016 ceph-mon[29101]: pgmap v512: 292 pgs: 292 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-26T04:24:54.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:54 smithi016 ceph-mon[29101]: osdmap e369: 8 total, 8 up, 8 in 2024-03-26T04:24:55.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:55 smithi183 ceph-mon[33745]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-26T04:24:55.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:55 smithi016 ceph-mon[29101]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-26T04:24:55.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:55 smithi016 ceph-mon[37215]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-26T04:24:56.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:56 smithi183 ceph-mon[33745]: pgmap v514: 292 pgs: 292 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-26T04:24:56.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:56 smithi016 ceph-mon[29101]: pgmap v514: 292 pgs: 292 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-26T04:24:56.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:56 smithi016 ceph-mon[37215]: pgmap v514: 292 pgs: 292 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-26T04:24:58.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:24:58 smithi183 ceph-mon[33745]: pgmap v515: 292 pgs: 292 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-26T04:24:58.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:24:58 smithi016 ceph-mon[29101]: pgmap v515: 292 pgs: 292 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-26T04:24:58.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:24:58 smithi016 ceph-mon[37215]: pgmap v515: 292 pgs: 292 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-26T04:24:58.913 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:24:58 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:24:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:25:00.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:00 smithi183 ceph-mon[33745]: pgmap v516: 292 pgs: 292 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T04:25:00.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:00 smithi016 ceph-mon[37215]: pgmap v516: 292 pgs: 292 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T04:25:00.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:00 smithi016 ceph-mon[29101]: pgmap v516: 292 pgs: 292 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T04:25:02.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:02 smithi183 ceph-mon[33745]: pgmap v517: 292 pgs: 292 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 593 B/s rd, 0 B/s wr, 0 op/s 2024-03-26T04:25:02.792 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:25:02 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:25:02.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:02 smithi016 ceph-mon[29101]: pgmap v517: 292 pgs: 292 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 593 B/s rd, 0 B/s wr, 0 op/s 2024-03-26T04:25:02.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:02 smithi016 ceph-mon[37215]: pgmap v517: 292 pgs: 292 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 593 B/s rd, 0 B/s wr, 0 op/s 2024-03-26T04:25:03.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:03 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:25:03.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:03 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:25:03.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:03 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:25:03.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:03 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:25:03.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:03 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:25:03.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:03 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:25:03.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:03 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:25:03.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:03 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:25:03.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:03 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:25:04.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:04 smithi016 ceph-mon[37215]: pgmap v518: 292 pgs: 292 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:25:04.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:04 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:25:04.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:04 smithi016 ceph-mon[37215]: osdmap e370: 8 total, 8 up, 8 in 2024-03-26T04:25:04.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:04 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-51"}]: dispatch 2024-03-26T04:25:04.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:04 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-51"}]: dispatch 2024-03-26T04:25:04.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:04 smithi016 ceph-mon[29101]: pgmap v518: 292 pgs: 292 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:25:04.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:04 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:25:04.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:04 smithi016 ceph-mon[29101]: osdmap e370: 8 total, 8 up, 8 in 2024-03-26T04:25:04.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:04 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-51"}]: dispatch 2024-03-26T04:25:04.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:04 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-51"}]: dispatch 2024-03-26T04:25:05.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:04 smithi183 ceph-mon[33745]: pgmap v518: 292 pgs: 292 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:25:05.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:04 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:25:05.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:04 smithi183 ceph-mon[33745]: osdmap e370: 8 total, 8 up, 8 in 2024-03-26T04:25:05.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:04 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-51"}]: dispatch 2024-03-26T04:25:05.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:04 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-51"}]: dispatch 2024-03-26T04:25:05.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:05 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-51"}]': finished 2024-03-26T04:25:05.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:05 smithi016 ceph-mon[37215]: osdmap e371: 8 total, 8 up, 8 in 2024-03-26T04:25:05.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:05 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:25:05.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:05 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:25:05.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:05 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-51"}]: dispatch 2024-03-26T04:25:05.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:05 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-51"}]: dispatch 2024-03-26T04:25:05.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:05 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-51"}]': finished 2024-03-26T04:25:05.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:05 smithi016 ceph-mon[29101]: osdmap e371: 8 total, 8 up, 8 in 2024-03-26T04:25:05.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:05 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:25:05.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:05 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:25:05.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:05 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-51"}]: dispatch 2024-03-26T04:25:05.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:05 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-51"}]: dispatch 2024-03-26T04:25:06.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:05 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-51"}]': finished 2024-03-26T04:25:06.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:05 smithi183 ceph-mon[33745]: osdmap e371: 8 total, 8 up, 8 in 2024-03-26T04:25:06.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:05 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:25:06.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:05 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:25:06.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:05 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-51"}]: dispatch 2024-03-26T04:25:06.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:05 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-51"}]: dispatch 2024-03-26T04:25:06.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:06 smithi016 ceph-mon[37215]: pgmap v521: 292 pgs: 292 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 0 op/s 2024-03-26T04:25:06.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:06 smithi016 ceph-mon[37215]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-26T04:25:06.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:06 smithi016 ceph-mon[37215]: osdmap e372: 8 total, 8 up, 8 in 2024-03-26T04:25:06.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:06 smithi016 ceph-mon[29101]: pgmap v521: 292 pgs: 292 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 0 op/s 2024-03-26T04:25:06.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:06 smithi016 ceph-mon[29101]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-26T04:25:06.915 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:06 smithi016 ceph-mon[29101]: osdmap e372: 8 total, 8 up, 8 in 2024-03-26T04:25:07.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:06 smithi183 ceph-mon[33745]: pgmap v521: 292 pgs: 292 active+clean; 8.4 MiB data, 650 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 0 op/s 2024-03-26T04:25:07.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:06 smithi183 ceph-mon[33745]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-26T04:25:07.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:06 smithi183 ceph-mon[33745]: osdmap e372: 8 total, 8 up, 8 in 2024-03-26T04:25:07.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:07 smithi016 ceph-mon[29101]: osdmap e373: 8 total, 8 up, 8 in 2024-03-26T04:25:07.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:07 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:25:07.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:07 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:25:07.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:07 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:25:07.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:07 smithi016 ceph-mon[37215]: osdmap e373: 8 total, 8 up, 8 in 2024-03-26T04:25:07.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:07 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:25:07.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:07 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:25:07.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:07 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:25:08.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:07 smithi183 ceph-mon[33745]: osdmap e373: 8 total, 8 up, 8 in 2024-03-26T04:25:08.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:07 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:25:08.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:07 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-53","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:25:08.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:07 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:25:08.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:25:08 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:25:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:25:09.010 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:08 smithi016 ceph-mon[37215]: pgmap v524: 292 pgs: 16 creating+peering, 16 unknown, 260 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T04:25:09.010 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:08 smithi016 ceph-mon[37215]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:25:09.010 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:08 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:25:09.010 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:08 smithi016 ceph-mon[37215]: osdmap e374: 8 total, 8 up, 8 in 2024-03-26T04:25:09.010 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:08 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:25:09.010 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:08 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:25:09.011 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:08 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-53"}]: dispatch 2024-03-26T04:25:09.011 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:08 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-53"}]: dispatch 2024-03-26T04:25:09.011 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:08 smithi016 ceph-mon[29101]: pgmap v524: 292 pgs: 16 creating+peering, 16 unknown, 260 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T04:25:09.011 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:08 smithi016 ceph-mon[29101]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:25:09.011 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:08 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:25:09.011 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:08 smithi016 ceph-mon[29101]: osdmap e374: 8 total, 8 up, 8 in 2024-03-26T04:25:09.011 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:08 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:25:09.011 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:08 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:25:09.011 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:08 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-53"}]: dispatch 2024-03-26T04:25:09.011 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:08 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-53"}]: dispatch 2024-03-26T04:25:09.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:08 smithi183 ceph-mon[33745]: pgmap v524: 292 pgs: 16 creating+peering, 16 unknown, 260 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T04:25:09.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:08 smithi183 ceph-mon[33745]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:25:09.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:08 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-53","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:25:09.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:08 smithi183 ceph-mon[33745]: osdmap e374: 8 total, 8 up, 8 in 2024-03-26T04:25:09.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:08 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:25:09.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:08 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:25:09.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:08 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-53"}]: dispatch 2024-03-26T04:25:09.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:08 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-53"}]: dispatch 2024-03-26T04:25:09.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:09 smithi016 ceph-mon[37215]: osdmap e375: 8 total, 8 up, 8 in 2024-03-26T04:25:09.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:09 smithi016 ceph-mon[29101]: osdmap e375: 8 total, 8 up, 8 in 2024-03-26T04:25:10.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:09 smithi183 ceph-mon[33745]: osdmap e375: 8 total, 8 up, 8 in 2024-03-26T04:25:10.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:10 smithi016 ceph-mon[29101]: pgmap v527: 260 pgs: 260 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T04:25:10.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:10 smithi016 ceph-mon[29101]: osdmap e376: 8 total, 8 up, 8 in 2024-03-26T04:25:10.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:10 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:25:10.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:10 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:25:10.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:10 smithi016 ceph-mon[37215]: pgmap v527: 260 pgs: 260 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T04:25:10.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:10 smithi016 ceph-mon[37215]: osdmap e376: 8 total, 8 up, 8 in 2024-03-26T04:25:10.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:10 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:25:10.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:10 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:25:11.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:10 smithi183 ceph-mon[33745]: pgmap v527: 260 pgs: 260 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T04:25:11.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:10 smithi183 ceph-mon[33745]: osdmap e376: 8 total, 8 up, 8 in 2024-03-26T04:25:11.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:10 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:25:11.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:10 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-55","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:25:11.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:11 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:25:11.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:11 smithi016 ceph-mon[37215]: osdmap e377: 8 total, 8 up, 8 in 2024-03-26T04:25:11.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:11 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:25:11.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:11 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:25:11.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:11 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:25:11.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:11 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-55"}]: dispatch 2024-03-26T04:25:11.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:11 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-55"}]: dispatch 2024-03-26T04:25:11.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:11 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:25:11.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:11 smithi016 ceph-mon[29101]: osdmap e377: 8 total, 8 up, 8 in 2024-03-26T04:25:11.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:11 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:25:11.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:11 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:25:11.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:11 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:25:11.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:11 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-55"}]: dispatch 2024-03-26T04:25:11.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:11 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-55"}]: dispatch 2024-03-26T04:25:12.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:11 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-55","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:25:12.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:11 smithi183 ceph-mon[33745]: osdmap e377: 8 total, 8 up, 8 in 2024-03-26T04:25:12.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:11 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:25:12.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:11 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:25:12.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:11 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:25:12.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:11 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-55"}]: dispatch 2024-03-26T04:25:12.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:11 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-55"}]: dispatch 2024-03-26T04:25:12.602 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:25:12 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:25:12.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:12 smithi016 ceph-mon[29101]: pgmap v530: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:25:12.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:12 smithi016 ceph-mon[29101]: osdmap e378: 8 total, 8 up, 8 in 2024-03-26T04:25:12.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:12 smithi016 ceph-mon[37215]: pgmap v530: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:25:12.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:12 smithi016 ceph-mon[37215]: osdmap e378: 8 total, 8 up, 8 in 2024-03-26T04:25:13.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:12 smithi183 ceph-mon[33745]: pgmap v530: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 651 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:25:13.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:12 smithi183 ceph-mon[33745]: osdmap e378: 8 total, 8 up, 8 in 2024-03-26T04:25:13.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:13 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:25:13.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:13 smithi016 ceph-mon[29101]: osdmap e379: 8 total, 8 up, 8 in 2024-03-26T04:25:13.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:13 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:25:13.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:13 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:25:13.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:13 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:25:13.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:13 smithi016 ceph-mon[37215]: osdmap e379: 8 total, 8 up, 8 in 2024-03-26T04:25:13.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:13 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:25:13.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:13 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:25:14.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:13 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:25:14.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:13 smithi183 ceph-mon[33745]: osdmap e379: 8 total, 8 up, 8 in 2024-03-26T04:25:14.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:13 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:25:14.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:13 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-57","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:25:14.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:14 smithi016 ceph-mon[29101]: pgmap v533: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-26T04:25:14.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:14 smithi016 ceph-mon[29101]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:25:14.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:14 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:25:14.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:14 smithi016 ceph-mon[29101]: osdmap e380: 8 total, 8 up, 8 in 2024-03-26T04:25:14.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:14 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:25:14.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:14 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:25:14.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:14 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-57"}]: dispatch 2024-03-26T04:25:14.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:14 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-57"}]: dispatch 2024-03-26T04:25:14.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:14 smithi016 ceph-mon[37215]: pgmap v533: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-26T04:25:14.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:14 smithi016 ceph-mon[37215]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:25:14.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:14 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:25:14.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:14 smithi016 ceph-mon[37215]: osdmap e380: 8 total, 8 up, 8 in 2024-03-26T04:25:14.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:14 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:25:14.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:14 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:25:14.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:14 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-57"}]: dispatch 2024-03-26T04:25:14.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:14 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-57"}]: dispatch 2024-03-26T04:25:15.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:14 smithi183 ceph-mon[33745]: pgmap v533: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-26T04:25:15.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:14 smithi183 ceph-mon[33745]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:25:15.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:14 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-57","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:25:15.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:14 smithi183 ceph-mon[33745]: osdmap e380: 8 total, 8 up, 8 in 2024-03-26T04:25:15.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:14 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:25:15.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:14 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:25:15.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:14 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-57"}]: dispatch 2024-03-26T04:25:15.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:14 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-57"}]: dispatch 2024-03-26T04:25:15.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:15 smithi016 ceph-mon[37215]: osdmap e381: 8 total, 8 up, 8 in 2024-03-26T04:25:15.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:15 smithi016 ceph-mon[29101]: osdmap e381: 8 total, 8 up, 8 in 2024-03-26T04:25:16.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:15 smithi183 ceph-mon[33745]: osdmap e381: 8 total, 8 up, 8 in 2024-03-26T04:25:16.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:16 smithi016 ceph-mon[37215]: pgmap v536: 260 pgs: 260 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-26T04:25:16.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:16 smithi016 ceph-mon[37215]: osdmap e382: 8 total, 8 up, 8 in 2024-03-26T04:25:16.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:16 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:25:16.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:16 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:25:16.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:16 smithi016 ceph-mon[29101]: pgmap v536: 260 pgs: 260 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-26T04:25:16.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:16 smithi016 ceph-mon[29101]: osdmap e382: 8 total, 8 up, 8 in 2024-03-26T04:25:16.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:16 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:25:16.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:16 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:25:17.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:16 smithi183 ceph-mon[33745]: pgmap v536: 260 pgs: 260 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-26T04:25:17.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:16 smithi183 ceph-mon[33745]: osdmap e382: 8 total, 8 up, 8 in 2024-03-26T04:25:17.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:16 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:25:17.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:16 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-59","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:25:17.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:17 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:25:17.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:17 smithi016 ceph-mon[29101]: osdmap e383: 8 total, 8 up, 8 in 2024-03-26T04:25:17.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:17 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:25:17.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:17 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:25:17.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:17 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:25:17.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:17 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-59"}]: dispatch 2024-03-26T04:25:17.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:17 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-59"}]: dispatch 2024-03-26T04:25:17.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:17 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:25:17.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:17 smithi016 ceph-mon[37215]: osdmap e383: 8 total, 8 up, 8 in 2024-03-26T04:25:17.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:17 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:25:17.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:17 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:25:17.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:17 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:25:17.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:17 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-59"}]: dispatch 2024-03-26T04:25:17.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:17 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-59"}]: dispatch 2024-03-26T04:25:18.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:17 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-59","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:25:18.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:17 smithi183 ceph-mon[33745]: osdmap e383: 8 total, 8 up, 8 in 2024-03-26T04:25:18.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:17 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:25:18.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:17 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:25:18.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:17 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:25:18.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:17 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-59"}]: dispatch 2024-03-26T04:25:18.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:17 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-59"}]: dispatch 2024-03-26T04:25:18.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:25:18 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:25:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:25:18.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:18 smithi016 ceph-mon[37215]: pgmap v539: 292 pgs: 19 unknown, 273 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-26T04:25:18.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:18 smithi016 ceph-mon[37215]: osdmap e384: 8 total, 8 up, 8 in 2024-03-26T04:25:18.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:18 smithi016 ceph-mon[29101]: pgmap v539: 292 pgs: 19 unknown, 273 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-26T04:25:18.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:18 smithi016 ceph-mon[29101]: osdmap e384: 8 total, 8 up, 8 in 2024-03-26T04:25:19.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:18 smithi183 ceph-mon[33745]: pgmap v539: 292 pgs: 19 unknown, 273 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-26T04:25:19.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:18 smithi183 ceph-mon[33745]: osdmap e384: 8 total, 8 up, 8 in 2024-03-26T04:25:19.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:19 smithi016 ceph-mon[29101]: osdmap e385: 8 total, 8 up, 8 in 2024-03-26T04:25:19.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:19 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:25:19.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:19 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:25:19.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:19 smithi016 ceph-mon[37215]: osdmap e385: 8 total, 8 up, 8 in 2024-03-26T04:25:19.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:19 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:25:19.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:19 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:25:20.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:19 smithi183 ceph-mon[33745]: osdmap e385: 8 total, 8 up, 8 in 2024-03-26T04:25:20.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:19 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:25:20.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:19 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-61","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:25:20.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:20 smithi016 ceph-mon[29101]: pgmap v542: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-26T04:25:20.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:20 smithi016 ceph-mon[29101]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:25:20.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:20 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:25:20.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:20 smithi016 ceph-mon[29101]: osdmap e386: 8 total, 8 up, 8 in 2024-03-26T04:25:20.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:20 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:25:20.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:20 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:25:20.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:20 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:25:20.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:20 smithi016 ceph-mon[37215]: pgmap v542: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-26T04:25:20.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:20 smithi016 ceph-mon[37215]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:25:20.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:20 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:25:20.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:20 smithi016 ceph-mon[37215]: osdmap e386: 8 total, 8 up, 8 in 2024-03-26T04:25:20.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:20 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:25:20.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:20 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:25:20.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:20 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:25:21.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:20 smithi183 ceph-mon[33745]: pgmap v542: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-26T04:25:21.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:20 smithi183 ceph-mon[33745]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:25:21.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:20 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-61","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:25:21.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:20 smithi183 ceph-mon[33745]: osdmap e386: 8 total, 8 up, 8 in 2024-03-26T04:25:21.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:20 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:25:21.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:20 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:25:21.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:20 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:25:21.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:21 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:25:21.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:21 smithi016 ceph-mon[37215]: osdmap e387: 8 total, 8 up, 8 in 2024-03-26T04:25:21.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:21 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:25:21.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:21 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:25:21.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:21 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-61"}]: dispatch 2024-03-26T04:25:21.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:21 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-61"}]: dispatch 2024-03-26T04:25:21.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:21 smithi016 ceph-mon[37215]: pgmap v545: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2024-03-26T04:25:21.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:21 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:25:21.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:21 smithi016 ceph-mon[29101]: osdmap e387: 8 total, 8 up, 8 in 2024-03-26T04:25:21.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:21 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:25:21.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:21 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:25:21.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:21 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-61"}]: dispatch 2024-03-26T04:25:21.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:21 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-61"}]: dispatch 2024-03-26T04:25:21.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:21 smithi016 ceph-mon[29101]: pgmap v545: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2024-03-26T04:25:22.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:21 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:25:22.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:21 smithi183 ceph-mon[33745]: osdmap e387: 8 total, 8 up, 8 in 2024-03-26T04:25:22.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:21 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:25:22.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:21 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:25:22.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:21 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-61"}]: dispatch 2024-03-26T04:25:22.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:21 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-61"}]: dispatch 2024-03-26T04:25:22.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:21 smithi183 ceph-mon[33745]: pgmap v545: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 511 B/s wr, 0 op/s 2024-03-26T04:25:22.662 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:25:22 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:25:22.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:22 smithi016 ceph-mon[37215]: osdmap e388: 8 total, 8 up, 8 in 2024-03-26T04:25:22.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:22 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:25:22.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:22 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:25:22.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:22 smithi016 ceph-mon[29101]: osdmap e388: 8 total, 8 up, 8 in 2024-03-26T04:25:22.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:22 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:25:22.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:22 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:25:23.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:22 smithi183 ceph-mon[33745]: osdmap e388: 8 total, 8 up, 8 in 2024-03-26T04:25:23.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:22 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:25:23.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:22 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:25:24.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:23 smithi183 ceph-mon[33745]: osdmap e389: 8 total, 8 up, 8 in 2024-03-26T04:25:24.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:23 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:25:24.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:23 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:25:24.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:23 smithi183 ceph-mon[33745]: pgmap v548: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:25:24.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:23 smithi016 ceph-mon[37215]: osdmap e389: 8 total, 8 up, 8 in 2024-03-26T04:25:24.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:23 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:25:24.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:23 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:25:24.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:23 smithi016 ceph-mon[37215]: pgmap v548: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:25:24.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:23 smithi016 ceph-mon[29101]: osdmap e389: 8 total, 8 up, 8 in 2024-03-26T04:25:24.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:23 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:25:24.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:23 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-63","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:25:24.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:23 smithi016 ceph-mon[29101]: pgmap v548: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:25:25.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:24 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:25:25.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:24 smithi183 ceph-mon[33745]: osdmap e390: 8 total, 8 up, 8 in 2024-03-26T04:25:25.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:24 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:25:25.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:24 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:25:25.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:24 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:25:25.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:24 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:25:25.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:24 smithi016 ceph-mon[37215]: osdmap e390: 8 total, 8 up, 8 in 2024-03-26T04:25:25.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:24 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:25:25.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:24 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:25:25.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:24 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:25:25.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:24 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-63","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:25:25.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:24 smithi016 ceph-mon[29101]: osdmap e390: 8 total, 8 up, 8 in 2024-03-26T04:25:25.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:24 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:25:25.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:24 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:25:25.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:24 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:25:26.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:25 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:25:26.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:25 smithi183 ceph-mon[33745]: osdmap e391: 8 total, 8 up, 8 in 2024-03-26T04:25:26.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:25 smithi183 ceph-mon[33745]: pgmap v551: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:25:26.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:25 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:25:26.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:25 smithi016 ceph-mon[29101]: osdmap e391: 8 total, 8 up, 8 in 2024-03-26T04:25:26.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:25 smithi016 ceph-mon[29101]: pgmap v551: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:25:26.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:25 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:25:26.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:25 smithi016 ceph-mon[37215]: osdmap e391: 8 total, 8 up, 8 in 2024-03-26T04:25:26.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:25 smithi016 ceph-mon[37215]: pgmap v551: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:25:27.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:26 smithi183 ceph-mon[33745]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:25:27.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:26 smithi183 ceph-mon[33745]: osdmap e392: 8 total, 8 up, 8 in 2024-03-26T04:25:27.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:26 smithi016 ceph-mon[37215]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:25:27.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:26 smithi016 ceph-mon[37215]: osdmap e392: 8 total, 8 up, 8 in 2024-03-26T04:25:27.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:26 smithi016 ceph-mon[29101]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:25:27.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:26 smithi016 ceph-mon[29101]: osdmap e392: 8 total, 8 up, 8 in 2024-03-26T04:25:28.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:27 smithi183 ceph-mon[33745]: osdmap e393: 8 total, 8 up, 8 in 2024-03-26T04:25:28.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:27 smithi183 ceph-mon[33745]: pgmap v554: 292 pgs: 15 unknown, 277 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-26T04:25:28.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:27 smithi016 ceph-mon[29101]: osdmap e393: 8 total, 8 up, 8 in 2024-03-26T04:25:28.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:27 smithi016 ceph-mon[29101]: pgmap v554: 292 pgs: 15 unknown, 277 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-26T04:25:28.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:27 smithi016 ceph-mon[37215]: osdmap e393: 8 total, 8 up, 8 in 2024-03-26T04:25:28.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:27 smithi016 ceph-mon[37215]: pgmap v554: 292 pgs: 15 unknown, 277 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-26T04:25:29.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:28 smithi183 ceph-mon[33745]: osdmap e394: 8 total, 8 up, 8 in 2024-03-26T04:25:29.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:28 smithi016 ceph-mon[37215]: osdmap e394: 8 total, 8 up, 8 in 2024-03-26T04:25:29.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:28 smithi016 ceph-mon[29101]: osdmap e394: 8 total, 8 up, 8 in 2024-03-26T04:25:29.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:25:28 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:25:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:25:30.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:29 smithi183 ceph-mon[33745]: osdmap e395: 8 total, 8 up, 8 in 2024-03-26T04:25:30.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:29 smithi183 ceph-mon[33745]: pgmap v557: 292 pgs: 292 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-03-26T04:25:30.063 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:29 smithi016 ceph-mon[37215]: osdmap e395: 8 total, 8 up, 8 in 2024-03-26T04:25:30.063 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:29 smithi016 ceph-mon[37215]: pgmap v557: 292 pgs: 292 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-03-26T04:25:30.064 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:29 smithi016 ceph-mon[29101]: osdmap e395: 8 total, 8 up, 8 in 2024-03-26T04:25:30.064 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:29 smithi016 ceph-mon[29101]: pgmap v557: 292 pgs: 292 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 2.2 KiB/s wr, 5 op/s 2024-03-26T04:25:32.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:32 smithi016 ceph-mon[37215]: pgmap v558: 292 pgs: 292 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-26T04:25:32.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:32 smithi016 ceph-mon[29101]: pgmap v558: 292 pgs: 292 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-26T04:25:32.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:32 smithi183 ceph-mon[33745]: pgmap v558: 292 pgs: 292 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.7 KiB/s wr, 4 op/s 2024-03-26T04:25:32.542 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:25:32 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:25:33.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:33 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:25:33.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:33 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:25:33.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:33 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:25:33.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:33 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:25:33.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:33 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:25:33.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:33 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:25:34.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:34 smithi016 ceph-mon[29101]: pgmap v559: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-03-26T04:25:34.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:34 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:25:34.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:34 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:25:34.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:34 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:25:34.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:34 smithi016 ceph-mon[37215]: pgmap v559: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-03-26T04:25:34.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:34 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:25:34.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:34 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:25:34.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:34 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:25:34.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:34 smithi183 ceph-mon[33745]: pgmap v559: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2024-03-26T04:25:34.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:34 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:25:34.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:34 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:25:34.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:34 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:25:36.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:36 smithi016 ceph-mon[37215]: pgmap v560: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 895 B/s wr, 2 op/s 2024-03-26T04:25:36.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:36 smithi016 ceph-mon[37215]: osdmap e396: 8 total, 8 up, 8 in 2024-03-26T04:25:36.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:36 smithi016 ceph-mon[29101]: pgmap v560: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 895 B/s wr, 2 op/s 2024-03-26T04:25:36.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:36 smithi016 ceph-mon[29101]: osdmap e396: 8 total, 8 up, 8 in 2024-03-26T04:25:36.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:36 smithi183 ceph-mon[33745]: pgmap v560: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 895 B/s wr, 2 op/s 2024-03-26T04:25:36.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:36 smithi183 ceph-mon[33745]: osdmap e396: 8 total, 8 up, 8 in 2024-03-26T04:25:38.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:38 smithi016 ceph-mon[37215]: pgmap v562: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 243 B/s wr, 2 op/s 2024-03-26T04:25:38.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:38 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:25:38.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:38 smithi016 ceph-mon[29101]: pgmap v562: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 243 B/s wr, 2 op/s 2024-03-26T04:25:38.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:38 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:25:38.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:38 smithi183 ceph-mon[33745]: pgmap v562: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 243 B/s wr, 2 op/s 2024-03-26T04:25:38.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:38 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:25:39.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:25:38 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:25:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:25:40.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:40 smithi183 ceph-mon[33745]: pgmap v563: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 204 B/s wr, 1 op/s 2024-03-26T04:25:40.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:40 smithi183 ceph-mon[33745]: osdmap e397: 8 total, 8 up, 8 in 2024-03-26T04:25:40.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:40 smithi016 ceph-mon[37215]: pgmap v563: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 204 B/s wr, 1 op/s 2024-03-26T04:25:40.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:40 smithi016 ceph-mon[37215]: osdmap e397: 8 total, 8 up, 8 in 2024-03-26T04:25:40.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:40 smithi016 ceph-mon[29101]: pgmap v563: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 204 B/s wr, 1 op/s 2024-03-26T04:25:40.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:40 smithi016 ceph-mon[29101]: osdmap e397: 8 total, 8 up, 8 in 2024-03-26T04:25:42.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:42 smithi183 ceph-mon[33745]: pgmap v565: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:25:42.542 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:25:42 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:25:42.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:42 smithi016 ceph-mon[29101]: pgmap v565: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:25:42.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:42 smithi016 ceph-mon[37215]: pgmap v565: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:25:43.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:43 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:25:43.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:43 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:25:43.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:43 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:25:44.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:44 smithi183 ceph-mon[33745]: pgmap v566: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:25:44.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:44 smithi016 ceph-mon[37215]: pgmap v566: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:25:44.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:44 smithi016 ceph-mon[29101]: pgmap v566: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:25:46.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:46 smithi183 ceph-mon[33745]: pgmap v567: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:25:46.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:46 smithi183 ceph-mon[33745]: osdmap e398: 8 total, 8 up, 8 in 2024-03-26T04:25:46.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:46 smithi016 ceph-mon[37215]: pgmap v567: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:25:46.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:46 smithi016 ceph-mon[37215]: osdmap e398: 8 total, 8 up, 8 in 2024-03-26T04:25:46.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:46 smithi016 ceph-mon[29101]: pgmap v567: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:25:46.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:46 smithi016 ceph-mon[29101]: osdmap e398: 8 total, 8 up, 8 in 2024-03-26T04:25:48.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:48 smithi183 ceph-mon[33745]: pgmap v569: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:25:48.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:48 smithi016 ceph-mon[37215]: pgmap v569: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:25:48.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:48 smithi016 ceph-mon[29101]: pgmap v569: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:25:49.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:25:48 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:25:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:25:50.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:50 smithi183 ceph-mon[33745]: pgmap v570: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:25:50.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:50 smithi183 ceph-mon[33745]: osdmap e399: 8 total, 8 up, 8 in 2024-03-26T04:25:50.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:50 smithi016 ceph-mon[37215]: pgmap v570: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:25:50.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:50 smithi016 ceph-mon[37215]: osdmap e399: 8 total, 8 up, 8 in 2024-03-26T04:25:50.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:50 smithi016 ceph-mon[29101]: pgmap v570: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:25:50.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:50 smithi016 ceph-mon[29101]: osdmap e399: 8 total, 8 up, 8 in 2024-03-26T04:25:52.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:52 smithi183 ceph-mon[33745]: pgmap v572: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:25:52.542 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:25:52 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:25:52.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:52 smithi016 ceph-mon[37215]: pgmap v572: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:25:52.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:52 smithi016 ceph-mon[29101]: pgmap v572: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:25:53.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:53 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:25:53.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:53 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:25:53.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:53 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:25:53.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:53 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:25:53.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:53 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:25:53.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:53 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:25:54.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:54 smithi183 ceph-mon[33745]: pgmap v573: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T04:25:54.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:54 smithi016 ceph-mon[29101]: pgmap v573: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T04:25:54.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:54 smithi016 ceph-mon[37215]: pgmap v573: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T04:25:56.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:56 smithi183 ceph-mon[33745]: pgmap v574: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:25:56.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:56 smithi183 ceph-mon[33745]: osdmap e400: 8 total, 8 up, 8 in 2024-03-26T04:25:56.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:56 smithi016 ceph-mon[37215]: pgmap v574: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:25:56.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:56 smithi016 ceph-mon[37215]: osdmap e400: 8 total, 8 up, 8 in 2024-03-26T04:25:56.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:56 smithi016 ceph-mon[29101]: pgmap v574: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:25:56.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:56 smithi016 ceph-mon[29101]: osdmap e400: 8 total, 8 up, 8 in 2024-03-26T04:25:58.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:25:58 smithi183 ceph-mon[33745]: pgmap v576: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T04:25:58.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:25:58 smithi016 ceph-mon[37215]: pgmap v576: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T04:25:58.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:25:58 smithi016 ceph-mon[29101]: pgmap v576: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T04:25:59.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:25:58 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:25:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:26:00.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:00 smithi183 ceph-mon[33745]: pgmap v577: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:26:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:00 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:26:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:00 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:26:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:00 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-63"}]: dispatch 2024-03-26T04:26:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:00 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-63"}]: dispatch 2024-03-26T04:26:00.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:00 smithi016 ceph-mon[37215]: pgmap v577: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:26:00.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:00 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:26:00.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:00 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:26:00.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:00 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-63"}]: dispatch 2024-03-26T04:26:00.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:00 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-63"}]: dispatch 2024-03-26T04:26:00.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:00 smithi016 ceph-mon[29101]: pgmap v577: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:26:00.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:00 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:26:00.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:00 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:26:00.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:00 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-63"}]: dispatch 2024-03-26T04:26:00.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:00 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-63"}]: dispatch 2024-03-26T04:26:01.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:01 smithi183 ceph-mon[33745]: osdmap e401: 8 total, 8 up, 8 in 2024-03-26T04:26:01.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:01 smithi016 ceph-mon[29101]: osdmap e401: 8 total, 8 up, 8 in 2024-03-26T04:26:01.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:01 smithi016 ceph-mon[37215]: osdmap e401: 8 total, 8 up, 8 in 2024-03-26T04:26:02.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:26:02 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:26:02.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:02 smithi183 ceph-mon[33745]: pgmap v579: 260 pgs: 260 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:26:02.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:02 smithi183 ceph-mon[33745]: osdmap e402: 8 total, 8 up, 8 in 2024-03-26T04:26:02.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:02 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:26:02.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:02 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:26:02.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:02 smithi016 ceph-mon[29101]: pgmap v579: 260 pgs: 260 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:26:02.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:02 smithi016 ceph-mon[29101]: osdmap e402: 8 total, 8 up, 8 in 2024-03-26T04:26:02.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:02 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:26:02.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:02 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:26:02.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:02 smithi016 ceph-mon[37215]: pgmap v579: 260 pgs: 260 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:26:02.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:02 smithi016 ceph-mon[37215]: osdmap e402: 8 total, 8 up, 8 in 2024-03-26T04:26:02.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:02 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:26:02.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:02 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-65","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:26:03.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:03 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:26:03.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:03 smithi183 ceph-mon[33745]: osdmap e403: 8 total, 8 up, 8 in 2024-03-26T04:26:03.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:03 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:26:03.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:03 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:26:03.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:03 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:26:03.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:03 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:26:03.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:03 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:26:03.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:03 smithi016 ceph-mon[37215]: osdmap e403: 8 total, 8 up, 8 in 2024-03-26T04:26:03.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:03 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:26:03.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:03 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:26:03.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:03 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:26:03.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:03 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:26:03.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:03 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-65","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:26:03.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:03 smithi016 ceph-mon[29101]: osdmap e403: 8 total, 8 up, 8 in 2024-03-26T04:26:03.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:03 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:26:03.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:03 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:26:03.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:03 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:26:03.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:03 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:26:04.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:04 smithi183 ceph-mon[33745]: pgmap v582: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:26:04.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:04 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:26:04.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:04 smithi183 ceph-mon[33745]: osdmap e404: 8 total, 8 up, 8 in 2024-03-26T04:26:04.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:04 smithi016 ceph-mon[37215]: pgmap v582: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:26:04.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:04 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:26:04.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:04 smithi016 ceph-mon[37215]: osdmap e404: 8 total, 8 up, 8 in 2024-03-26T04:26:04.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:04 smithi016 ceph-mon[29101]: pgmap v582: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:26:04.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:04 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:26:04.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:04 smithi016 ceph-mon[29101]: osdmap e404: 8 total, 8 up, 8 in 2024-03-26T04:26:05.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:05 smithi183 ceph-mon[33745]: osdmap e405: 8 total, 8 up, 8 in 2024-03-26T04:26:05.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:05 smithi016 ceph-mon[29101]: osdmap e405: 8 total, 8 up, 8 in 2024-03-26T04:26:05.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:05 smithi016 ceph-mon[37215]: osdmap e405: 8 total, 8 up, 8 in 2024-03-26T04:26:06.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:06 smithi183 ceph-mon[33745]: pgmap v585: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:26:06.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:06 smithi183 ceph-mon[33745]: osdmap e406: 8 total, 8 up, 8 in 2024-03-26T04:26:06.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:06 smithi016 ceph-mon[37215]: pgmap v585: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:26:06.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:06 smithi016 ceph-mon[37215]: osdmap e406: 8 total, 8 up, 8 in 2024-03-26T04:26:06.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:06 smithi016 ceph-mon[29101]: pgmap v585: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:26:06.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:06 smithi016 ceph-mon[29101]: osdmap e406: 8 total, 8 up, 8 in 2024-03-26T04:26:07.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:07 smithi183 ceph-mon[33745]: osdmap e407: 8 total, 8 up, 8 in 2024-03-26T04:26:07.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:07 smithi016 ceph-mon[29101]: osdmap e407: 8 total, 8 up, 8 in 2024-03-26T04:26:07.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:07 smithi016 ceph-mon[37215]: osdmap e407: 8 total, 8 up, 8 in 2024-03-26T04:26:08.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:08 smithi183 ceph-mon[33745]: pgmap v588: 292 pgs: 11 creating+peering, 281 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-03-26T04:26:08.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:08 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:26:08.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:08 smithi016 ceph-mon[37215]: pgmap v588: 292 pgs: 11 creating+peering, 281 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-03-26T04:26:08.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:08 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:26:08.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:08 smithi016 ceph-mon[29101]: pgmap v588: 292 pgs: 11 creating+peering, 281 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-03-26T04:26:08.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:08 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:26:09.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:26:08 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:26:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:26:10.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:10 smithi183 ceph-mon[33745]: pgmap v589: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-26T04:26:10.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:10 smithi016 ceph-mon[37215]: pgmap v589: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-26T04:26:10.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:10 smithi016 ceph-mon[29101]: pgmap v589: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.7 KiB/s wr, 6 op/s 2024-03-26T04:26:12.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:12 smithi016 ceph-mon[29101]: pgmap v590: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-26T04:26:12.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:12 smithi016 ceph-mon[29101]: osdmap e408: 8 total, 8 up, 8 in 2024-03-26T04:26:12.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:12 smithi016 ceph-mon[37215]: pgmap v590: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-26T04:26:12.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:12 smithi016 ceph-mon[37215]: osdmap e408: 8 total, 8 up, 8 in 2024-03-26T04:26:12.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:12 smithi183 ceph-mon[33745]: pgmap v590: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-03-26T04:26:12.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:12 smithi183 ceph-mon[33745]: osdmap e408: 8 total, 8 up, 8 in 2024-03-26T04:26:12.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:26:12 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:26:13.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:13 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:26:13.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:13 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:26:13.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:13 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:26:14.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:14 smithi016 ceph-mon[29101]: pgmap v592: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-03-26T04:26:14.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:14 smithi016 ceph-mon[37215]: pgmap v592: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-03-26T04:26:14.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:14 smithi183 ceph-mon[33745]: pgmap v592: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 1.3 KiB/s wr, 5 op/s 2024-03-26T04:26:16.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:16 smithi016 ceph-mon[37215]: pgmap v593: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2024-03-26T04:26:16.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:16 smithi016 ceph-mon[29101]: pgmap v593: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2024-03-26T04:26:16.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:16 smithi183 ceph-mon[33745]: pgmap v593: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2024-03-26T04:26:17.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:17 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:26:17.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:17 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:26:17.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:17 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-65"}]: dispatch 2024-03-26T04:26:17.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:17 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-65"}]: dispatch 2024-03-26T04:26:17.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:17 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:26:17.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:17 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:26:17.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:17 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-65"}]: dispatch 2024-03-26T04:26:17.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:17 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-65"}]: dispatch 2024-03-26T04:26:17.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:17 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:26:17.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:17 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:26:17.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:17 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-65"}]: dispatch 2024-03-26T04:26:17.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:17 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-65"}]: dispatch 2024-03-26T04:26:18.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:18 smithi016 ceph-mon[37215]: pgmap v594: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:26:18.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:18 smithi016 ceph-mon[37215]: osdmap e409: 8 total, 8 up, 8 in 2024-03-26T04:26:18.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:18 smithi016 ceph-mon[29101]: pgmap v594: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:26:18.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:18 smithi016 ceph-mon[29101]: osdmap e409: 8 total, 8 up, 8 in 2024-03-26T04:26:18.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:18 smithi183 ceph-mon[33745]: pgmap v594: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:26:18.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:18 smithi183 ceph-mon[33745]: osdmap e409: 8 total, 8 up, 8 in 2024-03-26T04:26:19.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:26:18 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:26:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:26:19.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:19 smithi183 ceph-mon[33745]: osdmap e410: 8 total, 8 up, 8 in 2024-03-26T04:26:19.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:19 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:26:19.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:19 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:26:19.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:19 smithi016 ceph-mon[37215]: osdmap e410: 8 total, 8 up, 8 in 2024-03-26T04:26:19.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:19 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:26:19.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:19 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:26:19.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:19 smithi016 ceph-mon[29101]: osdmap e410: 8 total, 8 up, 8 in 2024-03-26T04:26:19.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:19 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:26:19.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:19 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-67","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:26:20.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:20 smithi183 ceph-mon[33745]: pgmap v597: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:26:20.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:20 smithi183 ceph-mon[33745]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:26:20.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:20 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:26:20.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:20 smithi183 ceph-mon[33745]: osdmap e411: 8 total, 8 up, 8 in 2024-03-26T04:26:20.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:20 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:26:20.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:20 smithi016 ceph-mon[37215]: pgmap v597: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:26:20.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:20 smithi016 ceph-mon[37215]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:26:20.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:20 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:26:20.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:20 smithi016 ceph-mon[37215]: osdmap e411: 8 total, 8 up, 8 in 2024-03-26T04:26:20.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:20 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:26:20.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:20 smithi016 ceph-mon[29101]: pgmap v597: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:26:20.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:20 smithi016 ceph-mon[29101]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:26:20.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:20 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-67","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:26:20.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:20 smithi016 ceph-mon[29101]: osdmap e411: 8 total, 8 up, 8 in 2024-03-26T04:26:20.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:20 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:26:21.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:21 smithi183 ceph-mon[33745]: osdmap e412: 8 total, 8 up, 8 in 2024-03-26T04:26:21.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:21 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:26:21.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:21 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:26:21.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:21 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-67"}]: dispatch 2024-03-26T04:26:21.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:21 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-67"}]: dispatch 2024-03-26T04:26:21.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:21 smithi016 ceph-mon[37215]: osdmap e412: 8 total, 8 up, 8 in 2024-03-26T04:26:21.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:21 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:26:21.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:21 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:26:21.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:21 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-67"}]: dispatch 2024-03-26T04:26:21.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:21 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-67"}]: dispatch 2024-03-26T04:26:21.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:21 smithi016 ceph-mon[29101]: osdmap e412: 8 total, 8 up, 8 in 2024-03-26T04:26:21.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:21 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:26:21.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:21 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:26:21.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:21 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-67"}]: dispatch 2024-03-26T04:26:21.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:21 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-67"}]: dispatch 2024-03-26T04:26:22.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:22 smithi183 ceph-mon[33745]: pgmap v600: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:26:22.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:22 smithi183 ceph-mon[33745]: osdmap e413: 8 total, 8 up, 8 in 2024-03-26T04:26:22.542 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:26:22 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:26:22.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:22 smithi016 ceph-mon[37215]: pgmap v600: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:26:22.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:22 smithi016 ceph-mon[37215]: osdmap e413: 8 total, 8 up, 8 in 2024-03-26T04:26:22.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:22 smithi016 ceph-mon[29101]: pgmap v600: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:26:22.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:22 smithi016 ceph-mon[29101]: osdmap e413: 8 total, 8 up, 8 in 2024-03-26T04:26:23.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:23 smithi183 ceph-mon[33745]: osdmap e414: 8 total, 8 up, 8 in 2024-03-26T04:26:23.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:23 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:26:23.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:23 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:26:23.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:23 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:26:23.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:23 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:26:23.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:23 smithi016 ceph-mon[37215]: osdmap e414: 8 total, 8 up, 8 in 2024-03-26T04:26:23.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:23 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:26:23.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:23 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:26:23.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:23 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:26:23.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:23 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:26:23.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:23 smithi016 ceph-mon[29101]: osdmap e414: 8 total, 8 up, 8 in 2024-03-26T04:26:23.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:23 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:26:23.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:23 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-69","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:26:23.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:23 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:26:23.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:23 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:26:24.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:24 smithi183 ceph-mon[33745]: pgmap v603: 292 pgs: 16 creating+peering, 16 unknown, 260 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:26:24.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:24 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:26:24.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:24 smithi183 ceph-mon[33745]: osdmap e415: 8 total, 8 up, 8 in 2024-03-26T04:26:24.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:24 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:26:24.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:24 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:26:24.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:24 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:26:24.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:24 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-69"}]: dispatch 2024-03-26T04:26:24.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:24 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-69"}]: dispatch 2024-03-26T04:26:24.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:24 smithi016 ceph-mon[37215]: pgmap v603: 292 pgs: 16 creating+peering, 16 unknown, 260 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:26:24.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:24 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:26:24.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:24 smithi016 ceph-mon[37215]: osdmap e415: 8 total, 8 up, 8 in 2024-03-26T04:26:24.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:24 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:26:24.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:24 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:26:24.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:24 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:26:24.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:24 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-69"}]: dispatch 2024-03-26T04:26:24.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:24 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-69"}]: dispatch 2024-03-26T04:26:24.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:24 smithi016 ceph-mon[29101]: pgmap v603: 292 pgs: 16 creating+peering, 16 unknown, 260 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:26:24.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:24 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-69","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:26:24.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:24 smithi016 ceph-mon[29101]: osdmap e415: 8 total, 8 up, 8 in 2024-03-26T04:26:24.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:24 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:26:24.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:24 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:26:24.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:24 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:26:24.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:24 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-69"}]: dispatch 2024-03-26T04:26:24.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:24 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-69"}]: dispatch 2024-03-26T04:26:25.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:25 smithi183 ceph-mon[33745]: osdmap e416: 8 total, 8 up, 8 in 2024-03-26T04:26:25.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:25 smithi016 ceph-mon[37215]: osdmap e416: 8 total, 8 up, 8 in 2024-03-26T04:26:25.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:25 smithi016 ceph-mon[29101]: osdmap e416: 8 total, 8 up, 8 in 2024-03-26T04:26:26.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:26 smithi183 ceph-mon[33745]: pgmap v606: 260 pgs: 260 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:26:26.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:26 smithi183 ceph-mon[33745]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:26:26.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:26 smithi183 ceph-mon[33745]: osdmap e417: 8 total, 8 up, 8 in 2024-03-26T04:26:26.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:26 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:26:26.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:26 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:26:26.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:26 smithi016 ceph-mon[37215]: pgmap v606: 260 pgs: 260 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:26:26.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:26 smithi016 ceph-mon[37215]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:26:26.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:26 smithi016 ceph-mon[37215]: osdmap e417: 8 total, 8 up, 8 in 2024-03-26T04:26:26.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:26 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:26:26.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:26 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:26:26.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:26 smithi016 ceph-mon[29101]: pgmap v606: 260 pgs: 260 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:26:26.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:26 smithi016 ceph-mon[29101]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:26:26.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:26 smithi016 ceph-mon[29101]: osdmap e417: 8 total, 8 up, 8 in 2024-03-26T04:26:26.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:26 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:26:26.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:26 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-71","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:26:27.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:27 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:26:27.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:27 smithi183 ceph-mon[33745]: osdmap e418: 8 total, 8 up, 8 in 2024-03-26T04:26:27.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:27 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:26:27.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:27 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:26:27.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:27 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:26:27.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:27 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:26:27.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:27 smithi016 ceph-mon[29101]: osdmap e418: 8 total, 8 up, 8 in 2024-03-26T04:26:27.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:27 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:26:27.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:27 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:26:27.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:27 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:26:27.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:27 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-71","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:26:27.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:27 smithi016 ceph-mon[37215]: osdmap e418: 8 total, 8 up, 8 in 2024-03-26T04:26:27.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:27 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:26:27.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:27 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:26:27.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:27 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:26:28.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:28 smithi183 ceph-mon[33745]: pgmap v609: 292 pgs: 14 unknown, 278 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:26:28.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:28 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:26:28.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:28 smithi183 ceph-mon[33745]: osdmap e419: 8 total, 8 up, 8 in 2024-03-26T04:26:28.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:28 smithi016 ceph-mon[29101]: pgmap v609: 292 pgs: 14 unknown, 278 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:26:28.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:28 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:26:28.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:28 smithi016 ceph-mon[29101]: osdmap e419: 8 total, 8 up, 8 in 2024-03-26T04:26:28.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:28 smithi016 ceph-mon[37215]: pgmap v609: 292 pgs: 14 unknown, 278 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:26:28.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:28 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:26:28.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:28 smithi016 ceph-mon[37215]: osdmap e419: 8 total, 8 up, 8 in 2024-03-26T04:26:29.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:26:28 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:26:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:26:29.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:29 smithi183 ceph-mon[33745]: osdmap e420: 8 total, 8 up, 8 in 2024-03-26T04:26:29.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:29 smithi016 ceph-mon[29101]: osdmap e420: 8 total, 8 up, 8 in 2024-03-26T04:26:29.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:29 smithi016 ceph-mon[37215]: osdmap e420: 8 total, 8 up, 8 in 2024-03-26T04:26:30.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:30 smithi183 ceph-mon[33745]: pgmap v612: 292 pgs: 292 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:26:30.563 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:30 smithi016 ceph-mon[37215]: pgmap v612: 292 pgs: 292 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:26:30.563 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:30 smithi016 ceph-mon[29101]: pgmap v612: 292 pgs: 292 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:26:32.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:32 smithi183 ceph-mon[33745]: pgmap v613: 292 pgs: 292 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 519 B/s wr, 1 op/s 2024-03-26T04:26:32.542 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:26:32 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:26:32.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:32 smithi016 ceph-mon[37215]: pgmap v613: 292 pgs: 292 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 519 B/s wr, 1 op/s 2024-03-26T04:26:32.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:32 smithi016 ceph-mon[29101]: pgmap v613: 292 pgs: 292 active+clean; 8.4 MiB data, 645 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 519 B/s wr, 1 op/s 2024-03-26T04:26:33.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:33 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:26:33.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:33 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:26:33.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:33 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:26:34.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:34 smithi183 ceph-mon[33745]: pgmap v614: 292 pgs: 292 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.3 KiB/s wr, 5 op/s 2024-03-26T04:26:34.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:34 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:26:34.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:34 smithi016 ceph-mon[37215]: pgmap v614: 292 pgs: 292 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.3 KiB/s wr, 5 op/s 2024-03-26T04:26:34.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:34 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:26:34.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:34 smithi016 ceph-mon[29101]: pgmap v614: 292 pgs: 292 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 2.3 KiB/s wr, 5 op/s 2024-03-26T04:26:34.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:34 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:26:35.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:35 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:26:35.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:35 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:26:35.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:35 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:26:35.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:35 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:26:35.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:35 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:26:35.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:35 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:26:35.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:35 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:26:35.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:35 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:26:35.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:35 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:26:36.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:36 smithi016 ceph-mon[37215]: pgmap v615: 292 pgs: 292 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 2.0 KiB/s wr, 4 op/s 2024-03-26T04:26:36.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:36 smithi016 ceph-mon[29101]: pgmap v615: 292 pgs: 292 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 2.0 KiB/s wr, 4 op/s 2024-03-26T04:26:36.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:36 smithi183 ceph-mon[33745]: pgmap v615: 292 pgs: 292 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 895 B/s rd, 2.0 KiB/s wr, 4 op/s 2024-03-26T04:26:38.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:38 smithi016 ceph-mon[37215]: pgmap v616: 292 pgs: 292 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-03-26T04:26:38.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:38 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:26:38.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:38 smithi016 ceph-mon[29101]: pgmap v616: 292 pgs: 292 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-03-26T04:26:38.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:38 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:26:38.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:38 smithi183 ceph-mon[33745]: pgmap v616: 292 pgs: 292 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.6 KiB/s wr, 3 op/s 2024-03-26T04:26:38.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:38 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:26:39.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:26:38 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:26:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:26:40.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:40 smithi016 ceph-mon[37215]: pgmap v617: 292 pgs: 292 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-26T04:26:40.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:40 smithi016 ceph-mon[29101]: pgmap v617: 292 pgs: 292 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-26T04:26:40.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:40 smithi183 ceph-mon[33745]: pgmap v617: 292 pgs: 292 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-26T04:26:42.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:42 smithi016 ceph-mon[29101]: pgmap v618: 292 pgs: 292 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-03-26T04:26:42.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:42 smithi016 ceph-mon[37215]: pgmap v618: 292 pgs: 292 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-03-26T04:26:42.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:42 smithi183 ceph-mon[33745]: pgmap v618: 292 pgs: 292 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 1.1 KiB/s wr, 2 op/s 2024-03-26T04:26:42.792 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:26:42 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:26:43.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:43 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:26:43.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:43 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:26:43.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:43 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:26:44.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:44 smithi016 ceph-mon[37215]: pgmap v619: 292 pgs: 292 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-26T04:26:44.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:44 smithi016 ceph-mon[29101]: pgmap v619: 292 pgs: 292 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-26T04:26:44.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:44 smithi183 ceph-mon[33745]: pgmap v619: 292 pgs: 292 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.3 KiB/s wr, 4 op/s 2024-03-26T04:26:46.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:46 smithi016 ceph-mon[29101]: pgmap v620: 292 pgs: 292 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:26:46.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:46 smithi016 ceph-mon[37215]: pgmap v620: 292 pgs: 292 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:26:46.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:46 smithi183 ceph-mon[33745]: pgmap v620: 292 pgs: 292 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:26:48.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:48 smithi016 ceph-mon[37215]: pgmap v621: 292 pgs: 292 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:26:48.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:48 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:26:48.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:48 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:26:48.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:48 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-71"}]: dispatch 2024-03-26T04:26:48.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:48 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-71"}]: dispatch 2024-03-26T04:26:48.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:48 smithi016 ceph-mon[29101]: pgmap v621: 292 pgs: 292 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:26:48.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:48 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:26:48.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:48 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:26:48.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:48 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-71"}]: dispatch 2024-03-26T04:26:48.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:48 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-71"}]: dispatch 2024-03-26T04:26:48.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:48 smithi183 ceph-mon[33745]: pgmap v621: 292 pgs: 292 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:26:48.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:48 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:26:48.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:48 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:26:48.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:48 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-71"}]: dispatch 2024-03-26T04:26:48.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:48 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-71"}]: dispatch 2024-03-26T04:26:49.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:26:48 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:26:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:26:49.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:49 smithi016 ceph-mon[37215]: osdmap e421: 8 total, 8 up, 8 in 2024-03-26T04:26:49.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:49 smithi016 ceph-mon[29101]: osdmap e421: 8 total, 8 up, 8 in 2024-03-26T04:26:49.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:49 smithi183 ceph-mon[33745]: osdmap e421: 8 total, 8 up, 8 in 2024-03-26T04:26:50.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:50 smithi016 ceph-mon[29101]: pgmap v623: 260 pgs: 260 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-26T04:26:50.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:50 smithi016 ceph-mon[29101]: osdmap e422: 8 total, 8 up, 8 in 2024-03-26T04:26:50.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:50 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:26:50.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:50 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:26:50.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:50 smithi016 ceph-mon[37215]: pgmap v623: 260 pgs: 260 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-26T04:26:50.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:50 smithi016 ceph-mon[37215]: osdmap e422: 8 total, 8 up, 8 in 2024-03-26T04:26:50.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:50 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:26:50.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:50 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:26:50.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:50 smithi183 ceph-mon[33745]: pgmap v623: 260 pgs: 260 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-26T04:26:50.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:50 smithi183 ceph-mon[33745]: osdmap e422: 8 total, 8 up, 8 in 2024-03-26T04:26:50.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:50 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:26:50.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:50 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-73","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:26:51.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:51 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:26:51.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:51 smithi016 ceph-mon[29101]: osdmap e423: 8 total, 8 up, 8 in 2024-03-26T04:26:51.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:51 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:26:51.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:51 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:26:51.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:51 smithi016 ceph-mon[37215]: osdmap e423: 8 total, 8 up, 8 in 2024-03-26T04:26:51.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:51 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:26:51.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:51 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-73","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:26:51.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:51 smithi183 ceph-mon[33745]: osdmap e423: 8 total, 8 up, 8 in 2024-03-26T04:26:51.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:51 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:26:52.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:52 smithi183 ceph-mon[33745]: pgmap v626: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:26:52.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:52 smithi183 ceph-mon[33745]: osdmap e424: 8 total, 8 up, 8 in 2024-03-26T04:26:52.792 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:26:52 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:26:52.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:52 smithi016 ceph-mon[29101]: pgmap v626: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:26:52.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:52 smithi016 ceph-mon[29101]: osdmap e424: 8 total, 8 up, 8 in 2024-03-26T04:26:52.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:52 smithi016 ceph-mon[37215]: pgmap v626: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 629 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:26:52.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:52 smithi016 ceph-mon[37215]: osdmap e424: 8 total, 8 up, 8 in 2024-03-26T04:26:53.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:53 smithi183 ceph-mon[33745]: osdmap e425: 8 total, 8 up, 8 in 2024-03-26T04:26:53.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:53 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:26:53.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:53 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "287.d", "id": [7, 3]}]: dispatch 2024-03-26T04:26:53.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:53 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "287.d", "id": [7, 3]}]: dispatch 2024-03-26T04:26:53.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:53 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:26:53.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:53 smithi016 ceph-mon[29101]: osdmap e425: 8 total, 8 up, 8 in 2024-03-26T04:26:53.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:53 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:26:53.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:53 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "287.d", "id": [7, 3]}]: dispatch 2024-03-26T04:26:53.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:53 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "287.d", "id": [7, 3]}]: dispatch 2024-03-26T04:26:53.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:53 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:26:53.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:53 smithi016 ceph-mon[37215]: osdmap e425: 8 total, 8 up, 8 in 2024-03-26T04:26:53.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:53 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:26:53.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:53 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "287.d", "id": [7, 3]}]: dispatch 2024-03-26T04:26:53.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:53 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "287.d", "id": [7, 3]}]: dispatch 2024-03-26T04:26:53.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:53 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:26:54.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:54 smithi183 ceph-mon[33745]: pgmap v629: 292 pgs: 292 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T04:26:54.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:54 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "287.d", "id": [7, 3]}]': finished 2024-03-26T04:26:54.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:54 smithi183 ceph-mon[33745]: osdmap e426: 8 total, 8 up, 8 in 2024-03-26T04:26:54.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:54 smithi016 ceph-mon[29101]: pgmap v629: 292 pgs: 292 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T04:26:54.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:54 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "287.d", "id": [7, 3]}]': finished 2024-03-26T04:26:54.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:54 smithi016 ceph-mon[29101]: osdmap e426: 8 total, 8 up, 8 in 2024-03-26T04:26:54.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:54 smithi016 ceph-mon[37215]: pgmap v629: 292 pgs: 292 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T04:26:54.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:54 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "287.d", "id": [7, 3]}]': finished 2024-03-26T04:26:54.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:54 smithi016 ceph-mon[37215]: osdmap e426: 8 total, 8 up, 8 in 2024-03-26T04:26:55.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:55 smithi183 ceph-mon[33745]: osdmap e427: 8 total, 8 up, 8 in 2024-03-26T04:26:55.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:55 smithi016 ceph-mon[29101]: osdmap e427: 8 total, 8 up, 8 in 2024-03-26T04:26:55.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:55 smithi016 ceph-mon[37215]: osdmap e427: 8 total, 8 up, 8 in 2024-03-26T04:26:56.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:56 smithi183 ceph-mon[33745]: pgmap v632: 292 pgs: 292 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T04:26:56.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:56 smithi016 ceph-mon[29101]: pgmap v632: 292 pgs: 292 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T04:26:56.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:56 smithi016 ceph-mon[37215]: pgmap v632: 292 pgs: 292 active+clean; 8.4 MiB data, 633 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T04:26:58.745 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:26:58 smithi016 ceph-mon[37215]: pgmap v633: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 1.0 KiB/s wr, 5 op/s 2024-03-26T04:26:58.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:26:58 smithi016 ceph-mon[29101]: pgmap v633: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 1.0 KiB/s wr, 5 op/s 2024-03-26T04:26:58.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:26:58 smithi183 ceph-mon[33745]: pgmap v633: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 2.6 KiB/s rd, 1.0 KiB/s wr, 5 op/s 2024-03-26T04:26:59.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:26:58 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:26:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:27:00.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:00 smithi183 ceph-mon[33745]: pgmap v634: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 152 B/s wr, 3 op/s 2024-03-26T04:27:00.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:00 smithi016 ceph-mon[37215]: pgmap v634: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 152 B/s wr, 3 op/s 2024-03-26T04:27:00.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:00 smithi016 ceph-mon[29101]: pgmap v634: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 152 B/s wr, 3 op/s 2024-03-26T04:27:02.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:02 smithi183 ceph-mon[33745]: pgmap v635: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 127 B/s wr, 2 op/s 2024-03-26T04:27:02.792 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:27:02 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:27:02.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:02 smithi016 ceph-mon[37215]: pgmap v635: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 127 B/s wr, 2 op/s 2024-03-26T04:27:02.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:02 smithi016 ceph-mon[29101]: pgmap v635: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 127 B/s wr, 2 op/s 2024-03-26T04:27:03.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:03 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:27:03.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:03 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:03.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:03 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:03.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:03 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-73"}]: dispatch 2024-03-26T04:27:03.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:03 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-73"}]: dispatch 2024-03-26T04:27:03.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:03 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:27:03.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:03 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:03.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:03 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:03.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:03 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-73"}]: dispatch 2024-03-26T04:27:03.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:03 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-73"}]: dispatch 2024-03-26T04:27:03.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:03 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:27:03.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:03 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:03.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:03 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:03.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:03 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-73"}]: dispatch 2024-03-26T04:27:03.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:03 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-73"}]: dispatch 2024-03-26T04:27:04.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:04 smithi183 ceph-mon[33745]: pgmap v636: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 105 B/s wr, 2 op/s 2024-03-26T04:27:04.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:04 smithi183 ceph-mon[33745]: osdmap e428: 8 total, 8 up, 8 in 2024-03-26T04:27:04.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:04 smithi016 ceph-mon[37215]: pgmap v636: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 105 B/s wr, 2 op/s 2024-03-26T04:27:04.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:04 smithi016 ceph-mon[37215]: osdmap e428: 8 total, 8 up, 8 in 2024-03-26T04:27:04.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:04 smithi016 ceph-mon[29101]: pgmap v636: 292 pgs: 292 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 105 B/s wr, 2 op/s 2024-03-26T04:27:04.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:04 smithi016 ceph-mon[29101]: osdmap e428: 8 total, 8 up, 8 in 2024-03-26T04:27:05.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:05 smithi183 ceph-mon[33745]: osdmap e429: 8 total, 8 up, 8 in 2024-03-26T04:27:05.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:05 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:27:05.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:05 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:27:05.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:05 smithi016 ceph-mon[37215]: osdmap e429: 8 total, 8 up, 8 in 2024-03-26T04:27:05.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:05 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:27:05.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:05 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:27:05.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:05 smithi016 ceph-mon[29101]: osdmap e429: 8 total, 8 up, 8 in 2024-03-26T04:27:05.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:05 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:27:05.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:05 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-75","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:27:06.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:06 smithi183 ceph-mon[33745]: pgmap v639: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:27:06.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:06 smithi183 ceph-mon[33745]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:27:06.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:06 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:27:06.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:06 smithi183 ceph-mon[33745]: osdmap e430: 8 total, 8 up, 8 in 2024-03-26T04:27:06.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:06 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:27:06.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:06 smithi016 ceph-mon[29101]: pgmap v639: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:27:06.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:06 smithi016 ceph-mon[29101]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:27:06.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:06 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:27:06.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:06 smithi016 ceph-mon[29101]: osdmap e430: 8 total, 8 up, 8 in 2024-03-26T04:27:06.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:06 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:27:06.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:06 smithi016 ceph-mon[37215]: pgmap v639: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 634 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:27:06.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:06 smithi016 ceph-mon[37215]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:27:06.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:06 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-75","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:27:06.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:06 smithi016 ceph-mon[37215]: osdmap e430: 8 total, 8 up, 8 in 2024-03-26T04:27:06.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:06 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:27:07.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:07 smithi183 ceph-mon[33745]: osdmap e431: 8 total, 8 up, 8 in 2024-03-26T04:27:07.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:07 smithi016 ceph-mon[37215]: osdmap e431: 8 total, 8 up, 8 in 2024-03-26T04:27:07.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:07 smithi016 ceph-mon[29101]: osdmap e431: 8 total, 8 up, 8 in 2024-03-26T04:27:08.745 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:08 smithi016 ceph-mon[37215]: pgmap v642: 292 pgs: 16 unknown, 276 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:27:08.745 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:08 smithi016 ceph-mon[37215]: osdmap e432: 8 total, 8 up, 8 in 2024-03-26T04:27:08.745 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:08 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:08.745 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:08 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:08.745 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:08 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-75"}]: dispatch 2024-03-26T04:27:08.745 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:08 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-75"}]: dispatch 2024-03-26T04:27:08.745 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:08 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:27:08.745 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:08 smithi016 ceph-mon[29101]: pgmap v642: 292 pgs: 16 unknown, 276 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:27:08.745 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:08 smithi016 ceph-mon[29101]: osdmap e432: 8 total, 8 up, 8 in 2024-03-26T04:27:08.745 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:08 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:08.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:08 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:08.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:08 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-75"}]: dispatch 2024-03-26T04:27:08.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:08 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-75"}]: dispatch 2024-03-26T04:27:08.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:08 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:27:08.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:08 smithi183 ceph-mon[33745]: pgmap v642: 292 pgs: 16 unknown, 276 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:27:08.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:08 smithi183 ceph-mon[33745]: osdmap e432: 8 total, 8 up, 8 in 2024-03-26T04:27:08.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:08 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:08.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:08 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:08.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:08 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-75"}]: dispatch 2024-03-26T04:27:08.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:08 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-75"}]: dispatch 2024-03-26T04:27:08.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:08 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:27:09.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:27:08 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:27:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:27:09.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:09 smithi183 ceph-mon[33745]: osdmap e433: 8 total, 8 up, 8 in 2024-03-26T04:27:09.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:09 smithi016 ceph-mon[29101]: osdmap e433: 8 total, 8 up, 8 in 2024-03-26T04:27:09.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:09 smithi016 ceph-mon[37215]: osdmap e433: 8 total, 8 up, 8 in 2024-03-26T04:27:10.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:10 smithi183 ceph-mon[33745]: pgmap v645: 260 pgs: 260 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:27:10.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:10 smithi183 ceph-mon[33745]: osdmap e434: 8 total, 8 up, 8 in 2024-03-26T04:27:10.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:10 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:27:10.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:10 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:27:10.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:10 smithi016 ceph-mon[29101]: pgmap v645: 260 pgs: 260 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:27:10.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:10 smithi016 ceph-mon[29101]: osdmap e434: 8 total, 8 up, 8 in 2024-03-26T04:27:10.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:10 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:27:10.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:10 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:27:10.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:10 smithi016 ceph-mon[37215]: pgmap v645: 260 pgs: 260 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:27:10.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:10 smithi016 ceph-mon[37215]: osdmap e434: 8 total, 8 up, 8 in 2024-03-26T04:27:10.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:10 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:27:10.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:10 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-77","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:27:11.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:11 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:27:11.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:11 smithi183 ceph-mon[33745]: osdmap e435: 8 total, 8 up, 8 in 2024-03-26T04:27:11.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:11 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:27:11.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:11 smithi183 ceph-mon[33745]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:27:11.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:11 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:27:11.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:11 smithi016 ceph-mon[29101]: osdmap e435: 8 total, 8 up, 8 in 2024-03-26T04:27:11.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:11 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:27:11.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:11 smithi016 ceph-mon[29101]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:27:11.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:11 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-77","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:27:11.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:11 smithi016 ceph-mon[37215]: osdmap e435: 8 total, 8 up, 8 in 2024-03-26T04:27:11.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:11 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:27:11.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:11 smithi016 ceph-mon[37215]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:27:12.791 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:27:12 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:27:12.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:12 smithi183 ceph-mon[33745]: pgmap v648: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:27:12.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:12 smithi183 ceph-mon[33745]: osdmap e436: 8 total, 8 up, 8 in 2024-03-26T04:27:12.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:12 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:12.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:12 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:12.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:12 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-77"}]: dispatch 2024-03-26T04:27:12.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:12 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-77"}]: dispatch 2024-03-26T04:27:12.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:12 smithi016 ceph-mon[29101]: pgmap v648: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:27:12.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:12 smithi016 ceph-mon[29101]: osdmap e436: 8 total, 8 up, 8 in 2024-03-26T04:27:12.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:12 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:12.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:12 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:12.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:12 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-77"}]: dispatch 2024-03-26T04:27:12.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:12 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-77"}]: dispatch 2024-03-26T04:27:12.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:12 smithi016 ceph-mon[37215]: pgmap v648: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:27:12.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:12 smithi016 ceph-mon[37215]: osdmap e436: 8 total, 8 up, 8 in 2024-03-26T04:27:12.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:12 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:12.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:12 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:12.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:12 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-77"}]: dispatch 2024-03-26T04:27:12.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:12 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-77"}]: dispatch 2024-03-26T04:27:13.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:13 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:27:13.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:13 smithi183 ceph-mon[33745]: osdmap e437: 8 total, 8 up, 8 in 2024-03-26T04:27:13.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:13 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:27:13.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:13 smithi016 ceph-mon[37215]: osdmap e437: 8 total, 8 up, 8 in 2024-03-26T04:27:13.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:13 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:27:13.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:13 smithi016 ceph-mon[29101]: osdmap e437: 8 total, 8 up, 8 in 2024-03-26T04:27:14.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:14 smithi183 ceph-mon[33745]: pgmap v651: 260 pgs: 260 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-26T04:27:14.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:14 smithi183 ceph-mon[33745]: osdmap e438: 8 total, 8 up, 8 in 2024-03-26T04:27:14.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:14 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:27:14.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:14 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:27:14.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:14 smithi016 ceph-mon[29101]: pgmap v651: 260 pgs: 260 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-26T04:27:14.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:14 smithi016 ceph-mon[29101]: osdmap e438: 8 total, 8 up, 8 in 2024-03-26T04:27:14.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:14 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:27:14.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:14 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:27:14.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:14 smithi016 ceph-mon[37215]: pgmap v651: 260 pgs: 260 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-26T04:27:14.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:14 smithi016 ceph-mon[37215]: osdmap e438: 8 total, 8 up, 8 in 2024-03-26T04:27:14.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:14 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:27:14.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:14 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-79","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:27:15.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:15 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:27:15.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:15 smithi183 ceph-mon[33745]: osdmap e439: 8 total, 8 up, 8 in 2024-03-26T04:27:15.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:15 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:27:15.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:15 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:27:15.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:15 smithi016 ceph-mon[29101]: osdmap e439: 8 total, 8 up, 8 in 2024-03-26T04:27:15.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:15 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:27:15.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:15 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-79","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:27:15.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:15 smithi016 ceph-mon[37215]: osdmap e439: 8 total, 8 up, 8 in 2024-03-26T04:27:15.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:15 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:27:16.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:16 smithi183 ceph-mon[33745]: pgmap v654: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-26T04:27:16.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:16 smithi183 ceph-mon[33745]: osdmap e440: 8 total, 8 up, 8 in 2024-03-26T04:27:16.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:16 smithi016 ceph-mon[29101]: pgmap v654: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-26T04:27:16.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:16 smithi016 ceph-mon[29101]: osdmap e440: 8 total, 8 up, 8 in 2024-03-26T04:27:16.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:16 smithi016 ceph-mon[37215]: pgmap v654: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-26T04:27:16.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:16 smithi016 ceph-mon[37215]: osdmap e440: 8 total, 8 up, 8 in 2024-03-26T04:27:17.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:17 smithi183 ceph-mon[33745]: osdmap e441: 8 total, 8 up, 8 in 2024-03-26T04:27:17.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:17 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "290.7"}]: dispatch 2024-03-26T04:27:17.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:17 smithi183 ceph-mon[33745]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "290.7"}]: dispatch 2024-03-26T04:27:17.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:17 smithi016 ceph-mon[37215]: osdmap e441: 8 total, 8 up, 8 in 2024-03-26T04:27:17.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:17 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "290.7"}]: dispatch 2024-03-26T04:27:17.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:17 smithi016 ceph-mon[37215]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "290.7"}]: dispatch 2024-03-26T04:27:17.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:17 smithi016 ceph-mon[29101]: osdmap e441: 8 total, 8 up, 8 in 2024-03-26T04:27:17.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:17 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "pg deep-scrub", "pgid": "290.7"}]: dispatch 2024-03-26T04:27:17.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:17 smithi016 ceph-mon[29101]: from='mon.? v1:172.21.15.183:0/3428433360' entity='mon.' cmd=[{"prefix": "pg deep-scrub", "pgid": "290.7"}]: dispatch 2024-03-26T04:27:18.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:18 smithi183 ceph-mon[33745]: pgmap v657: 292 pgs: 10 unknown, 282 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:27:18.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:18 smithi183 ceph-mon[33745]: 290.7 deep-scrub starts 2024-03-26T04:27:18.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:18 smithi183 ceph-mon[33745]: 290.7 deep-scrub ok 2024-03-26T04:27:18.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:27:18 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:27:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:27:18.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:18 smithi016 ceph-mon[37215]: pgmap v657: 292 pgs: 10 unknown, 282 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:27:18.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:18 smithi016 ceph-mon[37215]: 290.7 deep-scrub starts 2024-03-26T04:27:18.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:18 smithi016 ceph-mon[37215]: 290.7 deep-scrub ok 2024-03-26T04:27:18.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:18 smithi016 ceph-mon[29101]: pgmap v657: 292 pgs: 10 unknown, 282 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:27:18.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:18 smithi016 ceph-mon[29101]: 290.7 deep-scrub starts 2024-03-26T04:27:18.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:18 smithi016 ceph-mon[29101]: 290.7 deep-scrub ok 2024-03-26T04:27:20.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:20 smithi183 ceph-mon[33745]: pgmap v658: 292 pgs: 292 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 910 B/s rd, 910 B/s wr, 2 op/s 2024-03-26T04:27:20.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:20 smithi016 ceph-mon[37215]: pgmap v658: 292 pgs: 292 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 910 B/s rd, 910 B/s wr, 2 op/s 2024-03-26T04:27:20.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:20 smithi016 ceph-mon[29101]: pgmap v658: 292 pgs: 292 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 910 B/s rd, 910 B/s wr, 2 op/s 2024-03-26T04:27:22.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:22 smithi183 ceph-mon[33745]: pgmap v659: 292 pgs: 292 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 773 B/s rd, 773 B/s wr, 2 op/s 2024-03-26T04:27:22.792 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:27:22 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:27:22.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:22 smithi016 ceph-mon[29101]: pgmap v659: 292 pgs: 292 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 773 B/s rd, 773 B/s wr, 2 op/s 2024-03-26T04:27:22.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:22 smithi016 ceph-mon[37215]: pgmap v659: 292 pgs: 292 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 773 B/s rd, 773 B/s wr, 2 op/s 2024-03-26T04:27:23.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:23 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:27:23.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:23 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:27:23.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:23 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:27:23.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:23 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:27:24.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:23 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:27:24.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:23 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:27:24.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:24 smithi016 ceph-mon[29101]: pgmap v660: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 639 B/s wr, 91 op/s 2024-03-26T04:27:24.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:24 smithi016 ceph-mon[37215]: pgmap v660: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 639 B/s wr, 91 op/s 2024-03-26T04:27:25.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:24 smithi183 ceph-mon[33745]: pgmap v660: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 639 B/s wr, 91 op/s 2024-03-26T04:27:26.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:26 smithi016 ceph-mon[29101]: pgmap v661: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 532 B/s wr, 76 op/s 2024-03-26T04:27:26.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:26 smithi016 ceph-mon[37215]: pgmap v661: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 532 B/s wr, 76 op/s 2024-03-26T04:27:27.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:26 smithi183 ceph-mon[33745]: pgmap v661: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s rd, 532 B/s wr, 76 op/s 2024-03-26T04:27:28.843 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:28 smithi183 ceph-mon[33745]: pgmap v662: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 482 B/s wr, 69 op/s 2024-03-26T04:27:28.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:28 smithi016 ceph-mon[37215]: pgmap v662: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 482 B/s wr, 69 op/s 2024-03-26T04:27:28.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:28 smithi016 ceph-mon[29101]: pgmap v662: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 482 B/s wr, 69 op/s 2024-03-26T04:27:28.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:27:28 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:27:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:27:30.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:30 smithi016 ceph-mon[37215]: pgmap v663: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 170 B/s wr, 60 op/s 2024-03-26T04:27:30.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:30 smithi016 ceph-mon[29101]: pgmap v663: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 170 B/s wr, 60 op/s 2024-03-26T04:27:31.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:30 smithi183 ceph-mon[33745]: pgmap v663: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 170 B/s wr, 60 op/s 2024-03-26T04:27:32.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:32 smithi183 ceph-mon[33745]: pgmap v664: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T04:27:32.792 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:27:32 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:27:32.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:32 smithi016 ceph-mon[37215]: pgmap v664: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T04:27:32.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:32 smithi016 ceph-mon[29101]: pgmap v664: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T04:27:33.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:33 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:27:33.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:33 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:27:34.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:33 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:27:34.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:34 smithi016 ceph-mon[37215]: pgmap v665: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T04:27:34.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:34 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:27:34.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:34 smithi016 ceph-mon[29101]: pgmap v665: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T04:27:34.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:34 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:27:35.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:34 smithi183 ceph-mon[33745]: pgmap v665: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T04:27:35.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:34 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:27:35.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:35 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:27:35.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:35 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:27:35.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:35 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:27:35.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:35 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:27:35.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:35 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:27:35.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:35 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:27:36.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:35 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:27:36.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:35 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:27:36.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:35 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:27:36.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:36 smithi016 ceph-mon[37215]: pgmap v666: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:27:36.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:36 smithi016 ceph-mon[29101]: pgmap v666: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:27:36.912 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 26 04:27:36 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-0[41205]: 2024-03-26T04:27:36.569+0000 7f08b344e700 -1 snap_mapper.add_oid found existing snaps mapped on 290:e717c37b:test-rados-api-smithi016-74534-80::foo:2, removing 2024-03-26T04:27:37.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:36 smithi183 ceph-mon[33745]: pgmap v666: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:27:37.042 INFO:journalctl@ceph.osd.4.smithi183.stdout:Mar 26 04:27:36 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-4[38631]: 2024-03-26T04:27:36.568+0000 7fad8ba87700 -1 snap_mapper.add_oid found existing snaps mapped on 290:e717c37b:test-rados-api-smithi016-74534-80::foo:2, removing 2024-03-26T04:27:37.042 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 26 04:27:36 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-5[44571]: 2024-03-26T04:27:36.568+0000 7fcff4345700 -1 snap_mapper.add_oid found existing snaps mapped on 290:e717c37b:test-rados-api-smithi016-74534-80::foo:2, removing 2024-03-26T04:27:37.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:37 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:37.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:37 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:37.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:37 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-79"}]: dispatch 2024-03-26T04:27:37.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:37 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-79"}]: dispatch 2024-03-26T04:27:37.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:37 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:37.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:37 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:37.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:37 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-79"}]: dispatch 2024-03-26T04:27:37.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:37 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-79"}]: dispatch 2024-03-26T04:27:38.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:37 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:38.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:37 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:38.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:37 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-79"}]: dispatch 2024-03-26T04:27:38.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:37 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-79"}]: dispatch 2024-03-26T04:27:38.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:38 smithi016 ceph-mon[29101]: pgmap v667: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 1 op/s 2024-03-26T04:27:38.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:38 smithi016 ceph-mon[29101]: osdmap e442: 8 total, 8 up, 8 in 2024-03-26T04:27:38.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:38 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:27:38.913 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:27:38 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:27:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:27:38.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:38 smithi016 ceph-mon[37215]: pgmap v667: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 1 op/s 2024-03-26T04:27:38.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:38 smithi016 ceph-mon[37215]: osdmap e442: 8 total, 8 up, 8 in 2024-03-26T04:27:38.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:38 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:27:39.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:38 smithi183 ceph-mon[33745]: pgmap v667: 292 pgs: 292 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 1 op/s 2024-03-26T04:27:39.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:38 smithi183 ceph-mon[33745]: osdmap e442: 8 total, 8 up, 8 in 2024-03-26T04:27:39.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:38 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:27:39.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:39 smithi016 ceph-mon[29101]: osdmap e443: 8 total, 8 up, 8 in 2024-03-26T04:27:39.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:39 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:27:39.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:39 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:27:39.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:39 smithi016 ceph-mon[37215]: osdmap e443: 8 total, 8 up, 8 in 2024-03-26T04:27:39.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:39 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:27:39.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:39 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:27:40.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:39 smithi183 ceph-mon[33745]: osdmap e443: 8 total, 8 up, 8 in 2024-03-26T04:27:40.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:39 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:27:40.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:39 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-81","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:27:40.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:40 smithi016 ceph-mon[29101]: pgmap v670: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-26T04:27:40.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:40 smithi016 ceph-mon[29101]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:27:40.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:40 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:27:40.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:40 smithi016 ceph-mon[29101]: osdmap e444: 8 total, 8 up, 8 in 2024-03-26T04:27:40.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:40 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:27:40.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:40 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:27:40.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:40 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:27:40.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:40 smithi016 ceph-mon[37215]: pgmap v670: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-26T04:27:40.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:40 smithi016 ceph-mon[37215]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:27:40.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:40 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:27:40.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:40 smithi016 ceph-mon[37215]: osdmap e444: 8 total, 8 up, 8 in 2024-03-26T04:27:40.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:40 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:27:40.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:40 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:27:40.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:40 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:27:41.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:40 smithi183 ceph-mon[33745]: pgmap v670: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1 op/s 2024-03-26T04:27:41.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:40 smithi183 ceph-mon[33745]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:27:41.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:40 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-81","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:27:41.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:40 smithi183 ceph-mon[33745]: osdmap e444: 8 total, 8 up, 8 in 2024-03-26T04:27:41.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:40 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:27:41.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:40 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:27:41.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:40 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:27:41.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:41 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:27:41.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:41 smithi016 ceph-mon[29101]: osdmap e445: 8 total, 8 up, 8 in 2024-03-26T04:27:41.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:41 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_tier","val": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:41.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:41 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_tier","val": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:41.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:41 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:27:41.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:41 smithi016 ceph-mon[37215]: osdmap e445: 8 total, 8 up, 8 in 2024-03-26T04:27:41.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:41 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_tier","val": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:41.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:41 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_tier","val": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:42.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:41 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:27:42.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:41 smithi183 ceph-mon[33745]: osdmap e445: 8 total, 8 up, 8 in 2024-03-26T04:27:42.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:41 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_tier","val": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:42.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:41 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_tier","val": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:42.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:42 smithi183 ceph-mon[33745]: pgmap v673: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T04:27:42.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:42 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_tier","val": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:27:42.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:42 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:27:42.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:42 smithi183 ceph-mon[33745]: osdmap e446: 8 total, 8 up, 8 in 2024-03-26T04:27:42.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:42 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:27:42.792 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:27:42 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:27:42.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:42 smithi016 ceph-mon[29101]: pgmap v673: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T04:27:42.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:42 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_tier","val": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:27:42.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:42 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:27:42.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:42 smithi016 ceph-mon[29101]: osdmap e446: 8 total, 8 up, 8 in 2024-03-26T04:27:42.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:42 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:27:42.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:42 smithi016 ceph-mon[37215]: pgmap v673: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T04:27:42.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:42 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_tier","val": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:27:42.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:42 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:27:42.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:42 smithi016 ceph-mon[37215]: osdmap e446: 8 total, 8 up, 8 in 2024-03-26T04:27:42.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:42 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:27:43.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:43 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:27:43.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:43 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T04:27:43.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:43 smithi016 ceph-mon[37215]: osdmap e447: 8 total, 8 up, 8 in 2024-03-26T04:27:43.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:43 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:27:43.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:43 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:27:43.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:43 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:27:43.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:43 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T04:27:43.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:43 smithi016 ceph-mon[29101]: osdmap e447: 8 total, 8 up, 8 in 2024-03-26T04:27:43.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:43 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:27:43.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:43 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:27:44.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:43 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:27:44.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:43 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T04:27:44.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:43 smithi183 ceph-mon[33745]: osdmap e447: 8 total, 8 up, 8 in 2024-03-26T04:27:44.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:43 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:27:44.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:43 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:27:44.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:44 smithi016 ceph-mon[37215]: pgmap v676: 292 pgs: 292 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:27:44.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:44 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T04:27:44.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:44 smithi016 ceph-mon[37215]: osdmap e448: 8 total, 8 up, 8 in 2024-03-26T04:27:44.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:44 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-26T04:27:44.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:44 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-26T04:27:44.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:44 smithi016 ceph-mon[29101]: pgmap v676: 292 pgs: 292 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:27:44.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:44 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T04:27:44.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:44 smithi016 ceph-mon[29101]: osdmap e448: 8 total, 8 up, 8 in 2024-03-26T04:27:44.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:44 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-26T04:27:44.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:44 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-26T04:27:45.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:44 smithi183 ceph-mon[33745]: pgmap v676: 292 pgs: 292 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:27:45.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:44 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T04:27:45.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:44 smithi183 ceph-mon[33745]: osdmap e448: 8 total, 8 up, 8 in 2024-03-26T04:27:45.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:44 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-26T04:27:45.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:44 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_cdc_chunk_size","val": "512"}]: dispatch 2024-03-26T04:27:45.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:45 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-03-26T04:27:45.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:45 smithi016 ceph-mon[37215]: osdmap e449: 8 total, 8 up, 8 in 2024-03-26T04:27:45.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:45 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-26T04:27:45.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:45 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-26T04:27:45.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:45 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-03-26T04:27:45.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:45 smithi016 ceph-mon[29101]: osdmap e449: 8 total, 8 up, 8 in 2024-03-26T04:27:45.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:45 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-26T04:27:45.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:45 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-26T04:27:46.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:45 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_cdc_chunk_size","val": "512"}]': finished 2024-03-26T04:27:46.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:45 smithi183 ceph-mon[33745]: osdmap e449: 8 total, 8 up, 8 in 2024-03-26T04:27:46.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:45 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-26T04:27:46.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:45 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_cdc_chunk_size","val": "16384"}]: dispatch 2024-03-26T04:27:46.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:46 smithi016 ceph-mon[37215]: pgmap v679: 292 pgs: 292 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:27:46.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:46 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-03-26T04:27:46.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:46 smithi016 ceph-mon[37215]: osdmap e450: 8 total, 8 up, 8 in 2024-03-26T04:27:46.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:46 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:27:46.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:46 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:27:46.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:46 smithi016 ceph-mon[37215]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:27:46.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:46 smithi016 ceph-mon[29101]: pgmap v679: 292 pgs: 292 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:27:46.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:46 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-03-26T04:27:46.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:46 smithi016 ceph-mon[29101]: osdmap e450: 8 total, 8 up, 8 in 2024-03-26T04:27:46.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:46 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:27:46.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:46 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:27:46.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:46 smithi016 ceph-mon[29101]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:27:47.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:46 smithi183 ceph-mon[33745]: pgmap v679: 292 pgs: 292 active+clean; 8.4 MiB data, 652 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:27:47.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:46 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_cdc_chunk_size","val": "16384"}]': finished 2024-03-26T04:27:47.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:46 smithi183 ceph-mon[33745]: osdmap e450: 8 total, 8 up, 8 in 2024-03-26T04:27:47.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:46 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:27:47.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:46 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:27:47.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:46 smithi183 ceph-mon[33745]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:27:47.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:47 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T04:27:47.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:47 smithi016 ceph-mon[29101]: osdmap e451: 8 total, 8 up, 8 in 2024-03-26T04:27:47.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:47 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:47.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:47 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:47.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:47 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-81"}]: dispatch 2024-03-26T04:27:47.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:47 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-81"}]: dispatch 2024-03-26T04:27:47.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:47 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T04:27:47.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:47 smithi016 ceph-mon[37215]: osdmap e451: 8 total, 8 up, 8 in 2024-03-26T04:27:47.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:47 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:47.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:47 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:47.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:47 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-81"}]: dispatch 2024-03-26T04:27:47.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:47 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-81"}]: dispatch 2024-03-26T04:27:48.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:47 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-81","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T04:27:48.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:47 smithi183 ceph-mon[33745]: osdmap e451: 8 total, 8 up, 8 in 2024-03-26T04:27:48.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:47 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:48.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:47 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:48.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:47 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-81"}]: dispatch 2024-03-26T04:27:48.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:47 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-81"}]: dispatch 2024-03-26T04:27:48.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:48 smithi016 ceph-mon[29101]: pgmap v682: 292 pgs: 292 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s rd, 7.5 KiB/s wr, 17 op/s 2024-03-26T04:27:48.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:48 smithi016 ceph-mon[29101]: osdmap e452: 8 total, 8 up, 8 in 2024-03-26T04:27:48.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:48 smithi016 ceph-mon[37215]: pgmap v682: 292 pgs: 292 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s rd, 7.5 KiB/s wr, 17 op/s 2024-03-26T04:27:48.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:48 smithi016 ceph-mon[37215]: osdmap e452: 8 total, 8 up, 8 in 2024-03-26T04:27:48.913 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:27:48 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:27:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:27:49.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:48 smithi183 ceph-mon[33745]: pgmap v682: 292 pgs: 292 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 4.5 KiB/s rd, 7.5 KiB/s wr, 17 op/s 2024-03-26T04:27:49.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:48 smithi183 ceph-mon[33745]: osdmap e452: 8 total, 8 up, 8 in 2024-03-26T04:27:49.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:49 smithi016 ceph-mon[29101]: osdmap e453: 8 total, 8 up, 8 in 2024-03-26T04:27:49.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:49 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:27:49.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:49 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:27:49.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:49 smithi016 ceph-mon[37215]: osdmap e453: 8 total, 8 up, 8 in 2024-03-26T04:27:49.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:49 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:27:49.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:49 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:27:50.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:49 smithi183 ceph-mon[33745]: osdmap e453: 8 total, 8 up, 8 in 2024-03-26T04:27:50.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:49 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:27:50.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:49 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-83","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:27:50.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:50 smithi016 ceph-mon[37215]: pgmap v685: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 12 KiB/s wr, 39 op/s 2024-03-26T04:27:50.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:50 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:27:50.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:50 smithi016 ceph-mon[37215]: osdmap e454: 8 total, 8 up, 8 in 2024-03-26T04:27:50.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:50 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:27:50.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:50 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:27:50.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:50 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:27:50.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:50 smithi016 ceph-mon[29101]: pgmap v685: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 12 KiB/s wr, 39 op/s 2024-03-26T04:27:50.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:50 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:27:50.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:50 smithi016 ceph-mon[29101]: osdmap e454: 8 total, 8 up, 8 in 2024-03-26T04:27:50.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:50 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:27:50.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:50 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:27:50.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:50 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:27:51.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:50 smithi183 ceph-mon[33745]: pgmap v685: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 12 KiB/s wr, 39 op/s 2024-03-26T04:27:51.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:50 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-83","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:27:51.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:50 smithi183 ceph-mon[33745]: osdmap e454: 8 total, 8 up, 8 in 2024-03-26T04:27:51.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:50 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:27:51.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:50 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:27:51.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:50 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-83","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:27:51.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:51 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:27:51.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:51 smithi016 ceph-mon[37215]: osdmap e455: 8 total, 8 up, 8 in 2024-03-26T04:27:51.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:51 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-83","var": "dedup_tier","val": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:51.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:51 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-83","var": "dedup_tier","val": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:51.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:51 smithi016 ceph-mon[37215]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:27:51.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:51 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:27:51.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:51 smithi016 ceph-mon[29101]: osdmap e455: 8 total, 8 up, 8 in 2024-03-26T04:27:51.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:51 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-83","var": "dedup_tier","val": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:51.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:51 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-83","var": "dedup_tier","val": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:51.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:51 smithi016 ceph-mon[29101]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:27:52.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:51 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-83","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:27:52.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:51 smithi183 ceph-mon[33745]: osdmap e455: 8 total, 8 up, 8 in 2024-03-26T04:27:52.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:51 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-83","var": "dedup_tier","val": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:52.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:51 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-83","var": "dedup_tier","val": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:52.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:51 smithi183 ceph-mon[33745]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:27:52.791 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:27:52 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:27:52.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:52 smithi183 ceph-mon[33745]: pgmap v688: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.7 KiB/s wr, 21 op/s 2024-03-26T04:27:52.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:52 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-83","var": "dedup_tier","val": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:27:52.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:52 smithi183 ceph-mon[33745]: osdmap e456: 8 total, 8 up, 8 in 2024-03-26T04:27:52.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:52 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:27:52.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:52 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:27:52.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:52 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:27:52.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:52 smithi016 ceph-mon[37215]: pgmap v688: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.7 KiB/s wr, 21 op/s 2024-03-26T04:27:52.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:52 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-83","var": "dedup_tier","val": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:27:52.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:52 smithi016 ceph-mon[37215]: osdmap e456: 8 total, 8 up, 8 in 2024-03-26T04:27:52.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:52 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:27:52.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:52 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:27:52.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:52 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:27:52.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:52 smithi016 ceph-mon[29101]: pgmap v688: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 653 MiB used, 715 GiB / 715 GiB avail; 5.5 KiB/s rd, 4.7 KiB/s wr, 21 op/s 2024-03-26T04:27:52.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:52 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-83","var": "dedup_tier","val": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:27:52.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:52 smithi016 ceph-mon[29101]: osdmap e456: 8 total, 8 up, 8 in 2024-03-26T04:27:52.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:52 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:27:52.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:52 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:27:52.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:52 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:27:54.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:53 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:27:54.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:53 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T04:27:54.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:53 smithi183 ceph-mon[33745]: osdmap e457: 8 total, 8 up, 8 in 2024-03-26T04:27:54.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:53 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:27:54.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:53 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:27:54.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:53 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:27:54.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:53 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T04:27:54.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:53 smithi016 ceph-mon[29101]: osdmap e457: 8 total, 8 up, 8 in 2024-03-26T04:27:54.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:53 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:27:54.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:53 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:27:54.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:53 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:27:54.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:53 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-83","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T04:27:54.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:53 smithi016 ceph-mon[37215]: osdmap e457: 8 total, 8 up, 8 in 2024-03-26T04:27:54.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:53 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:27:54.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:53 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-83","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:27:55.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:54 smithi183 ceph-mon[33745]: pgmap v691: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:27:55.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:54 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T04:27:55.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:54 smithi183 ceph-mon[33745]: osdmap e458: 8 total, 8 up, 8 in 2024-03-26T04:27:55.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:54 smithi016 ceph-mon[37215]: pgmap v691: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:27:55.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:54 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T04:27:55.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:54 smithi016 ceph-mon[37215]: osdmap e458: 8 total, 8 up, 8 in 2024-03-26T04:27:55.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:54 smithi016 ceph-mon[29101]: pgmap v691: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:27:55.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:54 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-83","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T04:27:55.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:54 smithi016 ceph-mon[29101]: osdmap e458: 8 total, 8 up, 8 in 2024-03-26T04:27:56.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:55 smithi183 ceph-mon[33745]: osdmap e459: 8 total, 8 up, 8 in 2024-03-26T04:27:56.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:55 smithi016 ceph-mon[37215]: osdmap e459: 8 total, 8 up, 8 in 2024-03-26T04:27:56.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:55 smithi016 ceph-mon[29101]: osdmap e459: 8 total, 8 up, 8 in 2024-03-26T04:27:57.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:56 smithi183 ceph-mon[33745]: pgmap v694: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:27:57.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:56 smithi183 ceph-mon[33745]: osdmap e460: 8 total, 8 up, 8 in 2024-03-26T04:27:57.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:56 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:57.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:56 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:57.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:56 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-83"}]: dispatch 2024-03-26T04:27:57.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:56 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-83"}]: dispatch 2024-03-26T04:27:57.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:56 smithi183 ceph-mon[33745]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:27:57.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:56 smithi016 ceph-mon[37215]: pgmap v694: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:27:57.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:56 smithi016 ceph-mon[37215]: osdmap e460: 8 total, 8 up, 8 in 2024-03-26T04:27:57.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:56 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:57.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:56 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:57.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:56 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-83"}]: dispatch 2024-03-26T04:27:57.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:56 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-83"}]: dispatch 2024-03-26T04:27:57.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:56 smithi016 ceph-mon[37215]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:27:57.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:56 smithi016 ceph-mon[29101]: pgmap v694: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:27:57.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:56 smithi016 ceph-mon[29101]: osdmap e460: 8 total, 8 up, 8 in 2024-03-26T04:27:57.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:56 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:57.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:56 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:27:57.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:56 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-83"}]: dispatch 2024-03-26T04:27:57.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:56 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-83"}]: dispatch 2024-03-26T04:27:57.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:56 smithi016 ceph-mon[29101]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:27:58.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:57 smithi183 ceph-mon[33745]: osdmap e461: 8 total, 8 up, 8 in 2024-03-26T04:27:58.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:57 smithi183 ceph-mon[33745]: pgmap v697: 260 pgs: 260 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-26T04:27:58.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:57 smithi016 ceph-mon[29101]: osdmap e461: 8 total, 8 up, 8 in 2024-03-26T04:27:58.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:57 smithi016 ceph-mon[29101]: pgmap v697: 260 pgs: 260 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-26T04:27:58.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:57 smithi016 ceph-mon[37215]: osdmap e461: 8 total, 8 up, 8 in 2024-03-26T04:27:58.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:57 smithi016 ceph-mon[37215]: pgmap v697: 260 pgs: 260 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-26T04:27:59.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:58 smithi183 ceph-mon[33745]: osdmap e462: 8 total, 8 up, 8 in 2024-03-26T04:27:59.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:27:59.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:58 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:27:59.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:27:58 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:27:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:27:59.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:58 smithi016 ceph-mon[37215]: osdmap e462: 8 total, 8 up, 8 in 2024-03-26T04:27:59.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:27:59.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:58 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:27:59.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:58 smithi016 ceph-mon[29101]: osdmap e462: 8 total, 8 up, 8 in 2024-03-26T04:27:59.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:27:59.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:58 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-85","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:28:00.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:59 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:28:00.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:59 smithi183 ceph-mon[33745]: osdmap e463: 8 total, 8 up, 8 in 2024-03-26T04:28:00.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:59 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:28:00.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:59 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:28:00.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:59 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:28:00.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:27:59 smithi183 ceph-mon[33745]: pgmap v700: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-03-26T04:28:00.064 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:59 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:28:00.064 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:59 smithi016 ceph-mon[29101]: osdmap e463: 8 total, 8 up, 8 in 2024-03-26T04:28:00.064 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:59 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:28:00.064 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:59 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:28:00.064 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:59 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:28:00.064 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:27:59 smithi016 ceph-mon[29101]: pgmap v700: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-03-26T04:28:00.064 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:59 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-85","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:28:00.065 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:59 smithi016 ceph-mon[37215]: osdmap e463: 8 total, 8 up, 8 in 2024-03-26T04:28:00.065 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:59 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:28:00.065 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:59 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:28:00.065 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:59 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-85","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:28:00.065 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:27:59 smithi016 ceph-mon[37215]: pgmap v700: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2024-03-26T04:28:01.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:00 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:28:01.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:00 smithi183 ceph-mon[33745]: osdmap e464: 8 total, 8 up, 8 in 2024-03-26T04:28:01.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:00 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-85","var": "dedup_tier","val": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:28:01.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:00 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-85","var": "dedup_tier","val": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:28:01.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:00 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:28:01.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:00 smithi016 ceph-mon[29101]: osdmap e464: 8 total, 8 up, 8 in 2024-03-26T04:28:01.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:00 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-85","var": "dedup_tier","val": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:28:01.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:00 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-85","var": "dedup_tier","val": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:28:01.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:00 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-85","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:28:01.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:00 smithi016 ceph-mon[37215]: osdmap e464: 8 total, 8 up, 8 in 2024-03-26T04:28:01.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:00 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-85","var": "dedup_tier","val": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:28:01.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:00 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-85","var": "dedup_tier","val": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:28:02.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:01 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-85","var": "dedup_tier","val": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:28:02.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:01 smithi183 ceph-mon[33745]: osdmap e465: 8 total, 8 up, 8 in 2024-03-26T04:28:02.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:01 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:28:02.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:01 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:28:02.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:01 smithi183 ceph-mon[33745]: pgmap v703: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 3.2 KiB/s wr, 6 op/s 2024-03-26T04:28:02.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:01 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T04:28:02.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:01 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-85","var": "dedup_tier","val": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:28:02.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:01 smithi016 ceph-mon[29101]: osdmap e465: 8 total, 8 up, 8 in 2024-03-26T04:28:02.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:01 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:28:02.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:01 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:28:02.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:01 smithi016 ceph-mon[29101]: pgmap v703: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 3.2 KiB/s wr, 6 op/s 2024-03-26T04:28:02.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:01 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T04:28:02.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:01 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-85","var": "dedup_tier","val": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:28:02.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:01 smithi016 ceph-mon[37215]: osdmap e465: 8 total, 8 up, 8 in 2024-03-26T04:28:02.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:01 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:28:02.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:01 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:28:02.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:01 smithi016 ceph-mon[37215]: pgmap v703: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 3.2 KiB/s wr, 6 op/s 2024-03-26T04:28:02.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:01 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-85","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T04:28:02.722 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:28:02 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:28:03.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:02 smithi183 ceph-mon[33745]: osdmap e466: 8 total, 8 up, 8 in 2024-03-26T04:28:03.091 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:02 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:28:03.092 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:02 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:28:03.092 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:02 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:28:03.092 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:02 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T04:28:03.092 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:02 smithi183 ceph-mon[33745]: osdmap e467: 8 total, 8 up, 8 in 2024-03-26T04:28:03.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:02 smithi016 ceph-mon[29101]: osdmap e466: 8 total, 8 up, 8 in 2024-03-26T04:28:03.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:02 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:28:03.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:02 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:28:03.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:02 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:28:03.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:02 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T04:28:03.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:02 smithi016 ceph-mon[29101]: osdmap e467: 8 total, 8 up, 8 in 2024-03-26T04:28:03.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:02 smithi016 ceph-mon[37215]: osdmap e466: 8 total, 8 up, 8 in 2024-03-26T04:28:03.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:02 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:28:03.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:02 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-85","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:28:03.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:02 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:28:03.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:02 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-85","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T04:28:03.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:02 smithi016 ceph-mon[37215]: osdmap e467: 8 total, 8 up, 8 in 2024-03-26T04:28:04.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:03 smithi183 ceph-mon[33745]: pgmap v706: 292 pgs: 292 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:28:04.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:03 smithi016 ceph-mon[37215]: pgmap v706: 292 pgs: 292 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:28:04.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:03 smithi016 ceph-mon[29101]: pgmap v706: 292 pgs: 292 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:28:05.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:04 smithi183 ceph-mon[33745]: osdmap e468: 8 total, 8 up, 8 in 2024-03-26T04:28:05.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:04 smithi016 ceph-mon[37215]: osdmap e468: 8 total, 8 up, 8 in 2024-03-26T04:28:05.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:04 smithi016 ceph-mon[29101]: osdmap e468: 8 total, 8 up, 8 in 2024-03-26T04:28:06.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:05 smithi183 ceph-mon[33745]: osdmap e469: 8 total, 8 up, 8 in 2024-03-26T04:28:06.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:05 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:28:06.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:05 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:28:06.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:05 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-85"}]: dispatch 2024-03-26T04:28:06.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:05 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-85"}]: dispatch 2024-03-26T04:28:06.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:05 smithi183 ceph-mon[33745]: pgmap v709: 292 pgs: 292 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:28:06.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:05 smithi016 ceph-mon[37215]: osdmap e469: 8 total, 8 up, 8 in 2024-03-26T04:28:06.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:05 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:28:06.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:05 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:28:06.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:05 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-85"}]: dispatch 2024-03-26T04:28:06.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:05 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-85"}]: dispatch 2024-03-26T04:28:06.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:05 smithi016 ceph-mon[37215]: pgmap v709: 292 pgs: 292 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:28:06.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:05 smithi016 ceph-mon[29101]: osdmap e469: 8 total, 8 up, 8 in 2024-03-26T04:28:06.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:05 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:28:06.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:05 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:28:06.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:05 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-85"}]: dispatch 2024-03-26T04:28:06.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:05 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-85"}]: dispatch 2024-03-26T04:28:06.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:05 smithi016 ceph-mon[29101]: pgmap v709: 292 pgs: 292 active+clean; 8.4 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:28:07.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:06 smithi183 ceph-mon[33745]: osdmap e470: 8 total, 8 up, 8 in 2024-03-26T04:28:07.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:06 smithi016 ceph-mon[29101]: osdmap e470: 8 total, 8 up, 8 in 2024-03-26T04:28:07.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:06 smithi016 ceph-mon[37215]: osdmap e470: 8 total, 8 up, 8 in 2024-03-26T04:28:07.761 INFO:tasks.workunit.client.0.smithi016.stdout: OK ] LibRadosTwoPoolsPP.ProxyRead (17822 ms) 2024-03-26T04:28:07.762 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-03-26T04:28:07.762 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (22137 ms) 2024-03-26T04:28:07.762 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-03-26T04:28:07.762 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (3021 ms) 2024-03-26T04:28:07.762 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-03-26T04:28:07.762 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T04:28:07.762 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (3023 ms) 2024-03-26T04:28:07.762 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-03-26T04:28:07.762 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (3013 ms) 2024-03-26T04:28:07.762 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-03-26T04:28:07.762 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T04:28:07.762 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (3025 ms) 2024-03-26T04:28:07.762 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-03-26T04:28:07.763 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T04:28:07.763 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (4022 ms) 2024-03-26T04:28:07.763 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-03-26T04:28:07.763 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T04:28:07.763 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (38551 ms) 2024-03-26T04:28:07.763 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-03-26T04:28:07.763 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T04:28:07.763 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (16941 ms) 2024-03-26T04:28:07.763 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-03-26T04:28:07.763 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T04:28:07.763 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (4025 ms) 2024-03-26T04:28:07.763 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-03-26T04:28:07.763 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T04:28:07.763 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (3015 ms) 2024-03-26T04:28:07.764 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-03-26T04:28:07.764 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T04:28:07.764 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (24193 ms) 2024-03-26T04:28:07.764 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-03-26T04:28:07.764 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T04:28:07.764 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (15062 ms) 2024-03-26T04:28:07.764 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-03-26T04:28:07.764 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T04:28:07.764 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (5028 ms) 2024-03-26T04:28:07.764 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-03-26T04:28:07.764 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T04:28:07.764 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (4031 ms) 2024-03-26T04:28:07.764 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-03-26T04:28:07.764 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T04:28:07.764 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: waiting for scrubs... 2024-03-26T04:28:07.765 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: done waiting 2024-03-26T04:28:07.765 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (25064 ms) 2024-03-26T04:28:07.765 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-03-26T04:28:07.765 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T04:28:07.765 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (10064 ms) 2024-03-26T04:28:07.765 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-03-26T04:28:07.765 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T04:28:07.765 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (9052 ms) 2024-03-26T04:28:07.765 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-03-26T04:28:07.765 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T04:28:07.765 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (9053 ms) 2024-03-26T04:28:07.765 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-03-26T04:28:08.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:07 smithi183 ceph-mon[33745]: osdmap e471: 8 total, 8 up, 8 in 2024-03-26T04:28:08.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:07 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:28:08.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:07 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:28:08.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:07 smithi183 ceph-mon[33745]: pgmap v712: 292 pgs: 20 creating+peering, 12 unknown, 260 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-26T04:28:08.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:07 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:28:08.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:07 smithi016 ceph-mon[37215]: osdmap e471: 8 total, 8 up, 8 in 2024-03-26T04:28:08.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:07 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:28:08.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:07 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:28:08.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:07 smithi016 ceph-mon[37215]: pgmap v712: 292 pgs: 20 creating+peering, 12 unknown, 260 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-26T04:28:08.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:07 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:28:08.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:07 smithi016 ceph-mon[29101]: osdmap e471: 8 total, 8 up, 8 in 2024-03-26T04:28:08.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:07 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:28:08.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:07 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-87","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:28:08.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:07 smithi016 ceph-mon[29101]: pgmap v712: 292 pgs: 20 creating+peering, 12 unknown, 260 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 4 op/s 2024-03-26T04:28:08.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:07 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:28:09.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:08 smithi183 ceph-mon[33745]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:28:09.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:08 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:28:09.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:08 smithi183 ceph-mon[33745]: osdmap e472: 8 total, 8 up, 8 in 2024-03-26T04:28:09.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:08 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:28:09.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:28:08 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:28:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:28:09.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:08 smithi016 ceph-mon[29101]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:28:09.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:08 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:28:09.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:08 smithi016 ceph-mon[29101]: osdmap e472: 8 total, 8 up, 8 in 2024-03-26T04:28:09.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:08 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:28:09.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:08 smithi016 ceph-mon[37215]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:28:09.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:08 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-87","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:28:09.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:08 smithi016 ceph-mon[37215]: osdmap e472: 8 total, 8 up, 8 in 2024-03-26T04:28:09.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:08 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:28:10.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:09 smithi183 ceph-mon[33745]: osdmap e473: 8 total, 8 up, 8 in 2024-03-26T04:28:10.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:09 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:28:10.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:09 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:28:10.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:09 smithi183 ceph-mon[33745]: pgmap v715: 324 pgs: 32 unknown, 32 creating+peering, 260 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-03-26T04:28:10.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:09 smithi016 ceph-mon[37215]: osdmap e473: 8 total, 8 up, 8 in 2024-03-26T04:28:10.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:09 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:28:10.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:09 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:28:10.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:09 smithi016 ceph-mon[37215]: pgmap v715: 324 pgs: 32 unknown, 32 creating+peering, 260 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-03-26T04:28:10.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:09 smithi016 ceph-mon[29101]: osdmap e473: 8 total, 8 up, 8 in 2024-03-26T04:28:10.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:09 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:28:10.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:09 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-87","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:28:10.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:09 smithi016 ceph-mon[29101]: pgmap v715: 324 pgs: 32 unknown, 32 creating+peering, 260 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 4.7 KiB/s wr, 11 op/s 2024-03-26T04:28:11.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:10 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:28:11.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:10 smithi183 ceph-mon[33745]: osdmap e474: 8 total, 8 up, 8 in 2024-03-26T04:28:11.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:10 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-87","var": "dedup_tier","val": "test-rados-api-smithi016-74534-89-test-flush"}]: dispatch 2024-03-26T04:28:11.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:10 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-87","var": "dedup_tier","val": "test-rados-api-smithi016-74534-89-test-flush"}]: dispatch 2024-03-26T04:28:11.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:10 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:28:11.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:10 smithi016 ceph-mon[37215]: osdmap e474: 8 total, 8 up, 8 in 2024-03-26T04:28:11.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:10 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-87","var": "dedup_tier","val": "test-rados-api-smithi016-74534-89-test-flush"}]: dispatch 2024-03-26T04:28:11.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:10 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-87","var": "dedup_tier","val": "test-rados-api-smithi016-74534-89-test-flush"}]: dispatch 2024-03-26T04:28:11.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:10 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-87","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:28:11.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:10 smithi016 ceph-mon[29101]: osdmap e474: 8 total, 8 up, 8 in 2024-03-26T04:28:11.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:10 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-87","var": "dedup_tier","val": "test-rados-api-smithi016-74534-89-test-flush"}]: dispatch 2024-03-26T04:28:11.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:10 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-87","var": "dedup_tier","val": "test-rados-api-smithi016-74534-89-test-flush"}]: dispatch 2024-03-26T04:28:12.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:11 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-87","var": "dedup_tier","val": "test-rados-api-smithi016-74534-89-test-flush"}]': finished 2024-03-26T04:28:12.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:11 smithi183 ceph-mon[33745]: osdmap e475: 8 total, 8 up, 8 in 2024-03-26T04:28:12.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:11 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:28:12.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:11 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:28:12.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:11 smithi183 ceph-mon[33745]: pgmap v718: 324 pgs: 32 unknown, 32 creating+peering, 260 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 3.2 KiB/s wr, 6 op/s 2024-03-26T04:28:12.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:11 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-87","var": "dedup_tier","val": "test-rados-api-smithi016-74534-89-test-flush"}]': finished 2024-03-26T04:28:12.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:11 smithi016 ceph-mon[29101]: osdmap e475: 8 total, 8 up, 8 in 2024-03-26T04:28:12.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:11 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:28:12.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:11 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:28:12.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:11 smithi016 ceph-mon[29101]: pgmap v718: 324 pgs: 32 unknown, 32 creating+peering, 260 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 3.2 KiB/s wr, 6 op/s 2024-03-26T04:28:12.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:11 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-87","var": "dedup_tier","val": "test-rados-api-smithi016-74534-89-test-flush"}]': finished 2024-03-26T04:28:12.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:11 smithi016 ceph-mon[37215]: osdmap e475: 8 total, 8 up, 8 in 2024-03-26T04:28:12.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:11 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:28:12.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:11 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:28:12.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:11 smithi016 ceph-mon[37215]: pgmap v718: 324 pgs: 32 unknown, 32 creating+peering, 260 active+clean; 8.4 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 3.2 KiB/s wr, 6 op/s 2024-03-26T04:28:12.774 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:28:12 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:28:13.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:12 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T04:28:13.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:12 smithi183 ceph-mon[33745]: osdmap e476: 8 total, 8 up, 8 in 2024-03-26T04:28:13.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:12 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:28:13.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:12 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:28:13.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:12 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:28:13.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:12 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T04:28:13.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:12 smithi016 ceph-mon[37215]: osdmap e476: 8 total, 8 up, 8 in 2024-03-26T04:28:13.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:12 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:28:13.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:12 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:28:13.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:12 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:28:13.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:12 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-87","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T04:28:13.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:12 smithi016 ceph-mon[29101]: osdmap e476: 8 total, 8 up, 8 in 2024-03-26T04:28:13.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:12 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:28:13.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:12 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-87","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:28:13.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:12 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:28:14.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:13 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T04:28:14.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:13 smithi183 ceph-mon[33745]: osdmap e477: 8 total, 8 up, 8 in 2024-03-26T04:28:14.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:13 smithi183 ceph-mon[33745]: pgmap v721: 324 pgs: 324 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:28:14.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:13 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T04:28:14.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:13 smithi016 ceph-mon[37215]: osdmap e477: 8 total, 8 up, 8 in 2024-03-26T04:28:14.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:13 smithi016 ceph-mon[37215]: pgmap v721: 324 pgs: 324 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:28:14.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:13 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-87","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T04:28:14.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:13 smithi016 ceph-mon[29101]: osdmap e477: 8 total, 8 up, 8 in 2024-03-26T04:28:14.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:13 smithi016 ceph-mon[29101]: pgmap v721: 324 pgs: 324 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:28:15.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:14 smithi183 ceph-mon[33745]: osdmap e478: 8 total, 8 up, 8 in 2024-03-26T04:28:15.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:14 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:28:15.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:14 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:28:15.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:14 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-87"}]: dispatch 2024-03-26T04:28:15.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:14 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-87"}]: dispatch 2024-03-26T04:28:15.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:14 smithi016 ceph-mon[37215]: osdmap e478: 8 total, 8 up, 8 in 2024-03-26T04:28:15.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:14 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:28:15.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:14 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:28:15.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:14 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-87"}]: dispatch 2024-03-26T04:28:15.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:14 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-87"}]: dispatch 2024-03-26T04:28:15.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:14 smithi016 ceph-mon[29101]: osdmap e478: 8 total, 8 up, 8 in 2024-03-26T04:28:15.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:14 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:28:15.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:14 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:28:15.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:14 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-87"}]: dispatch 2024-03-26T04:28:15.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:14 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-87"}]: dispatch 2024-03-26T04:28:16.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:15 smithi016 ceph-mon[29101]: osdmap e479: 8 total, 8 up, 8 in 2024-03-26T04:28:16.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:15 smithi016 ceph-mon[29101]: pgmap v724: 260 pgs: 260 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:28:16.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:15 smithi016 ceph-mon[37215]: osdmap e479: 8 total, 8 up, 8 in 2024-03-26T04:28:16.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:15 smithi016 ceph-mon[37215]: pgmap v724: 260 pgs: 260 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:28:16.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:15 smithi183 ceph-mon[33745]: osdmap e479: 8 total, 8 up, 8 in 2024-03-26T04:28:16.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:15 smithi183 ceph-mon[33745]: pgmap v724: 260 pgs: 260 active+clean; 8.4 MiB data, 661 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:28:17.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:16 smithi016 ceph-mon[29101]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:28:17.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:16 smithi016 ceph-mon[29101]: osdmap e480: 8 total, 8 up, 8 in 2024-03-26T04:28:17.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:16 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:28:17.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:16 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:28:17.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:16 smithi016 ceph-mon[37215]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:28:17.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:16 smithi016 ceph-mon[37215]: osdmap e480: 8 total, 8 up, 8 in 2024-03-26T04:28:17.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:16 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:28:17.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:16 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:28:17.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:16 smithi183 ceph-mon[33745]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:28:17.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:16 smithi183 ceph-mon[33745]: osdmap e480: 8 total, 8 up, 8 in 2024-03-26T04:28:17.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:16 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:28:17.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:16 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-90","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:28:18.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:17 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:28:18.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:17 smithi016 ceph-mon[37215]: osdmap e481: 8 total, 8 up, 8 in 2024-03-26T04:28:18.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:17 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:28:18.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:17 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:28:18.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:17 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:28:18.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:17 smithi016 ceph-mon[37215]: pgmap v727: 292 pgs: 12 unknown, 280 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:28:18.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:17 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:28:18.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:17 smithi016 ceph-mon[29101]: osdmap e481: 8 total, 8 up, 8 in 2024-03-26T04:28:18.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:17 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:28:18.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:17 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:28:18.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:17 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:28:18.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:17 smithi016 ceph-mon[29101]: pgmap v727: 292 pgs: 12 unknown, 280 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:28:18.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:17 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-90","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:28:18.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:17 smithi183 ceph-mon[33745]: osdmap e481: 8 total, 8 up, 8 in 2024-03-26T04:28:18.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:17 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:28:18.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:17 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:28:18.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:17 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-6","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:28:18.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:17 smithi183 ceph-mon[33745]: pgmap v727: 292 pgs: 12 unknown, 280 active+clean; 8.4 MiB data, 665 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:28:19.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:18 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:28:19.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:18 smithi016 ceph-mon[29101]: osdmap e482: 8 total, 8 up, 8 in 2024-03-26T04:28:19.163 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:28:18 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:28:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:28:19.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:18 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:28:19.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:18 smithi016 ceph-mon[37215]: osdmap e482: 8 total, 8 up, 8 in 2024-03-26T04:28:19.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:18 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-6","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:28:19.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:18 smithi183 ceph-mon[33745]: osdmap e482: 8 total, 8 up, 8 in 2024-03-26T04:28:20.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:19 smithi016 ceph-mon[29101]: osdmap e483: 8 total, 8 up, 8 in 2024-03-26T04:28:20.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:19 smithi016 ceph-mon[29101]: pgmap v730: 292 pgs: 292 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:28:20.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:19 smithi016 ceph-mon[37215]: osdmap e483: 8 total, 8 up, 8 in 2024-03-26T04:28:20.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:19 smithi016 ceph-mon[37215]: pgmap v730: 292 pgs: 292 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:28:20.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:19 smithi183 ceph-mon[33745]: osdmap e483: 8 total, 8 up, 8 in 2024-03-26T04:28:20.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:19 smithi183 ceph-mon[33745]: pgmap v730: 292 pgs: 292 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:28:21.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:21 smithi016 ceph-mon[29101]: osdmap e484: 8 total, 8 up, 8 in 2024-03-26T04:28:21.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:21 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:28:21.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:21 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:28:21.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:21 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-90"}]: dispatch 2024-03-26T04:28:21.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:21 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-90"}]: dispatch 2024-03-26T04:28:21.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:21 smithi016 ceph-mon[37215]: osdmap e484: 8 total, 8 up, 8 in 2024-03-26T04:28:21.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:21 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:28:21.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:21 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:28:21.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:21 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-90"}]: dispatch 2024-03-26T04:28:21.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:21 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-90"}]: dispatch 2024-03-26T04:28:21.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:21 smithi183 ceph-mon[33745]: osdmap e484: 8 total, 8 up, 8 in 2024-03-26T04:28:21.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:21 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:28:21.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:21 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:28:21.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:21 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-90"}]: dispatch 2024-03-26T04:28:21.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:21 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-90"}]: dispatch 2024-03-26T04:28:22.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:22 smithi016 ceph-mon[29101]: osdmap e485: 8 total, 8 up, 8 in 2024-03-26T04:28:22.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:22 smithi016 ceph-mon[29101]: pgmap v733: 260 pgs: 260 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:28:22.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:22 smithi016 ceph-mon[29101]: osdmap e486: 8 total, 8 up, 8 in 2024-03-26T04:28:22.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:22 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:28:22.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:22 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:28:22.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:22 smithi016 ceph-mon[37215]: osdmap e485: 8 total, 8 up, 8 in 2024-03-26T04:28:22.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:22 smithi016 ceph-mon[37215]: pgmap v733: 260 pgs: 260 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:28:22.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:22 smithi016 ceph-mon[37215]: osdmap e486: 8 total, 8 up, 8 in 2024-03-26T04:28:22.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:22 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:28:22.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:22 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:28:22.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:22 smithi183 ceph-mon[33745]: osdmap e485: 8 total, 8 up, 8 in 2024-03-26T04:28:22.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:22 smithi183 ceph-mon[33745]: pgmap v733: 260 pgs: 260 active+clean; 8.4 MiB data, 669 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:28:22.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:22 smithi183 ceph-mon[33745]: osdmap e486: 8 total, 8 up, 8 in 2024-03-26T04:28:22.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:22 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:28:22.489 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:22 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-92","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:28:22.794 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:28:22 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:28:23.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:23 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:28:23.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:23 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:28:23.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:23 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:28:23.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:23 smithi016 ceph-mon[29101]: osdmap e487: 8 total, 8 up, 8 in 2024-03-26T04:28:23.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:23 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:28:23.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:23 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:28:23.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:23 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:28:23.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:23 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:28:23.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:23 smithi016 ceph-mon[37215]: osdmap e487: 8 total, 8 up, 8 in 2024-03-26T04:28:23.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:23 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:28:23.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:23 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:28:23.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:23 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:28:23.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:23 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-92","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:28:23.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:23 smithi183 ceph-mon[33745]: osdmap e487: 8 total, 8 up, 8 in 2024-03-26T04:28:23.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:23 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:28:24.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:24 smithi016 ceph-mon[29101]: pgmap v736: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T04:28:24.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:24 smithi016 ceph-mon[29101]: osdmap e488: 8 total, 8 up, 8 in 2024-03-26T04:28:24.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:24 smithi016 ceph-mon[37215]: pgmap v736: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T04:28:24.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:24 smithi016 ceph-mon[37215]: osdmap e488: 8 total, 8 up, 8 in 2024-03-26T04:28:24.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:24 smithi183 ceph-mon[33745]: pgmap v736: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T04:28:24.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:24 smithi183 ceph-mon[33745]: osdmap e488: 8 total, 8 up, 8 in 2024-03-26T04:28:26.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:25 smithi016 ceph-mon[37215]: osdmap e489: 8 total, 8 up, 8 in 2024-03-26T04:28:26.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:25 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:28:26.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:25 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:28:26.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:25 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-92"}]: dispatch 2024-03-26T04:28:26.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:25 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-92"}]: dispatch 2024-03-26T04:28:26.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:25 smithi016 ceph-mon[37215]: pgmap v739: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T04:28:26.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:25 smithi016 ceph-mon[29101]: osdmap e489: 8 total, 8 up, 8 in 2024-03-26T04:28:26.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:25 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:28:26.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:25 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:28:26.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:25 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-92"}]: dispatch 2024-03-26T04:28:26.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:25 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-92"}]: dispatch 2024-03-26T04:28:26.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:25 smithi016 ceph-mon[29101]: pgmap v739: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T04:28:26.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:25 smithi183 ceph-mon[33745]: osdmap e489: 8 total, 8 up, 8 in 2024-03-26T04:28:26.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:25 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:28:26.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:25 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:28:26.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:25 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-92"}]: dispatch 2024-03-26T04:28:26.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:25 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-92"}]: dispatch 2024-03-26T04:28:26.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:25 smithi183 ceph-mon[33745]: pgmap v739: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 2 op/s 2024-03-26T04:28:27.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:26 smithi016 ceph-mon[29101]: osdmap e490: 8 total, 8 up, 8 in 2024-03-26T04:28:27.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:26 smithi016 ceph-mon[37215]: osdmap e490: 8 total, 8 up, 8 in 2024-03-26T04:28:27.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:26 smithi183 ceph-mon[33745]: osdmap e490: 8 total, 8 up, 8 in 2024-03-26T04:28:28.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:27 smithi016 ceph-mon[29101]: osdmap e491: 8 total, 8 up, 8 in 2024-03-26T04:28:28.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:27 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:28:28.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:27 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:28:28.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:27 smithi016 ceph-mon[29101]: pgmap v742: 292 pgs: 22 creating+peering, 10 unknown, 260 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:28:28.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:27 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:28:28.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:27 smithi016 ceph-mon[37215]: osdmap e491: 8 total, 8 up, 8 in 2024-03-26T04:28:28.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:27 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:28:28.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:27 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:28:28.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:27 smithi016 ceph-mon[37215]: pgmap v742: 292 pgs: 22 creating+peering, 10 unknown, 260 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:28:28.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:27 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:28:28.193 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:27 smithi183 ceph-mon[33745]: osdmap e491: 8 total, 8 up, 8 in 2024-03-26T04:28:28.193 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:27 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:28:28.193 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:27 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-94","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:28:28.193 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:27 smithi183 ceph-mon[33745]: pgmap v742: 292 pgs: 22 creating+peering, 10 unknown, 260 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:28:28.193 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:27 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-94","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:28:29.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:28:28 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:28:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:28:29.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:28 smithi016 ceph-mon[37215]: osdmap e492: 8 total, 8 up, 8 in 2024-03-26T04:28:29.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:28 smithi016 ceph-mon[37215]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:28:29.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:28 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:28:29.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:28 smithi016 ceph-mon[37215]: osdmap e493: 8 total, 8 up, 8 in 2024-03-26T04:28:29.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:28 smithi016 ceph-mon[29101]: osdmap e492: 8 total, 8 up, 8 in 2024-03-26T04:28:29.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:28 smithi016 ceph-mon[29101]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:28:29.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:28 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:28:29.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:28 smithi016 ceph-mon[29101]: osdmap e493: 8 total, 8 up, 8 in 2024-03-26T04:28:29.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:28 smithi183 ceph-mon[33745]: osdmap e492: 8 total, 8 up, 8 in 2024-03-26T04:28:29.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:28 smithi183 ceph-mon[33745]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:28:29.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:28 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:28:29.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:28 smithi183 ceph-mon[33745]: osdmap e493: 8 total, 8 up, 8 in 2024-03-26T04:28:30.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:29 smithi016 ceph-mon[37215]: pgmap v745: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:28:30.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:29 smithi016 ceph-mon[37215]: osdmap e494: 8 total, 8 up, 8 in 2024-03-26T04:28:30.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:29 smithi016 ceph-mon[29101]: pgmap v745: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:28:30.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:29 smithi016 ceph-mon[29101]: osdmap e494: 8 total, 8 up, 8 in 2024-03-26T04:28:30.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:29 smithi183 ceph-mon[33745]: pgmap v745: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:28:30.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:29 smithi183 ceph-mon[33745]: osdmap e494: 8 total, 8 up, 8 in 2024-03-26T04:28:32.496 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:32 smithi183 ceph-mon[33745]: pgmap v747: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 711 B/s wr, 2 op/s 2024-03-26T04:28:32.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:32 smithi016 ceph-mon[29101]: pgmap v747: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 711 B/s wr, 2 op/s 2024-03-26T04:28:32.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:32 smithi016 ceph-mon[37215]: pgmap v747: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 711 B/s wr, 2 op/s 2024-03-26T04:28:32.791 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:28:32 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:28:33.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:33 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:28:33.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:33 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:28:33.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:33 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:28:34.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:34 smithi183 ceph-mon[33745]: pgmap v748: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.8 KiB/s wr, 5 op/s 2024-03-26T04:28:34.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:34 smithi016 ceph-mon[29101]: pgmap v748: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.8 KiB/s wr, 5 op/s 2024-03-26T04:28:34.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:34 smithi016 ceph-mon[37215]: pgmap v748: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.8 KiB/s rd, 1.8 KiB/s wr, 5 op/s 2024-03-26T04:28:35.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:35 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:28:35.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:35 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:28:35.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:35 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:28:36.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:36 smithi183 ceph-mon[33745]: pgmap v749: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-26T04:28:36.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:36 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:36 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:36 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:36 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:36 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:36 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:36 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:36 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:36 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:36 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:36 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:36 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:36 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:36 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:36 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:36 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:36 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:28:36.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:36 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:28:36.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:36 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:28:36.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:36 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:28:36.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:36 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:28:36.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:36 smithi183 ceph-mon[33745]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:28:36.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[37215]: pgmap v749: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-26T04:28:36.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:28:36.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:28:36.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:28:36.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:28:36.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:28:36.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[37215]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:28:36.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[29101]: pgmap v749: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-26T04:28:36.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-26T04:28:36.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:28:36.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:28:36.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:28:36.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:28:36.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:28:36.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:36 smithi016 ceph-mon[29101]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:28:37.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:37 smithi183 ceph-mon[33745]: Adjusting osd_memory_target on smithi016 to 3249M 2024-03-26T04:28:37.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:37 smithi183 ceph-mon[33745]: Adjusting osd_memory_target on smithi183 to 3248M 2024-03-26T04:28:37.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:37 smithi016 ceph-mon[37215]: Adjusting osd_memory_target on smithi016 to 3249M 2024-03-26T04:28:37.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:37 smithi016 ceph-mon[37215]: Adjusting osd_memory_target on smithi183 to 3248M 2024-03-26T04:28:37.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:37 smithi016 ceph-mon[29101]: Adjusting osd_memory_target on smithi016 to 3249M 2024-03-26T04:28:37.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:37 smithi016 ceph-mon[29101]: Adjusting osd_memory_target on smithi183 to 3248M 2024-03-26T04:28:38.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:38 smithi183 ceph-mon[33745]: pgmap v750: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 986 B/s wr, 4 op/s 2024-03-26T04:28:38.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:38 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:28:38.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:38 smithi016 ceph-mon[37215]: pgmap v750: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 986 B/s wr, 4 op/s 2024-03-26T04:28:38.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:38 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:28:38.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:38 smithi016 ceph-mon[29101]: pgmap v750: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 986 B/s wr, 4 op/s 2024-03-26T04:28:38.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:38 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:28:39.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:28:38 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:28:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:28:40.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:40 smithi183 ceph-mon[33745]: pgmap v751: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 818 B/s wr, 3 op/s 2024-03-26T04:28:40.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:40 smithi016 ceph-mon[37215]: pgmap v751: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 818 B/s wr, 3 op/s 2024-03-26T04:28:40.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:40 smithi016 ceph-mon[29101]: pgmap v751: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 818 B/s wr, 3 op/s 2024-03-26T04:28:41.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:41 smithi183 ceph-mon[33745]: osdmap e495: 8 total, 8 up, 8 in 2024-03-26T04:28:41.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:41 smithi016 ceph-mon[29101]: osdmap e495: 8 total, 8 up, 8 in 2024-03-26T04:28:41.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:41 smithi016 ceph-mon[37215]: osdmap e495: 8 total, 8 up, 8 in 2024-03-26T04:28:42.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:42 smithi183 ceph-mon[33745]: pgmap v753: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 818 B/s wr, 3 op/s 2024-03-26T04:28:42.542 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:28:42 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:28:42.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:42 smithi016 ceph-mon[37215]: pgmap v753: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 818 B/s wr, 3 op/s 2024-03-26T04:28:42.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:42 smithi016 ceph-mon[29101]: pgmap v753: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 818 B/s wr, 3 op/s 2024-03-26T04:28:43.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:43 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:28:43.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:43 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:28:43.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:43 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:28:44.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:44 smithi183 ceph-mon[33745]: pgmap v754: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:28:44.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:44 smithi016 ceph-mon[37215]: pgmap v754: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:28:44.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:44 smithi016 ceph-mon[29101]: pgmap v754: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:28:46.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:46 smithi183 ceph-mon[33745]: pgmap v755: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:28:46.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:46 smithi183 ceph-mon[33745]: osdmap e496: 8 total, 8 up, 8 in 2024-03-26T04:28:46.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:46 smithi016 ceph-mon[37215]: pgmap v755: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:28:46.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:46 smithi016 ceph-mon[37215]: osdmap e496: 8 total, 8 up, 8 in 2024-03-26T04:28:46.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:46 smithi016 ceph-mon[29101]: pgmap v755: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:28:46.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:46 smithi016 ceph-mon[29101]: osdmap e496: 8 total, 8 up, 8 in 2024-03-26T04:28:48.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:48 smithi183 ceph-mon[33745]: pgmap v757: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:28:48.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:48 smithi016 ceph-mon[37215]: pgmap v757: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:28:48.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:48 smithi016 ceph-mon[29101]: pgmap v757: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:28:49.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:28:48 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:28:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:28:50.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:50 smithi183 ceph-mon[33745]: pgmap v758: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:28:50.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:50 smithi016 ceph-mon[37215]: pgmap v758: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:28:50.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:50 smithi016 ceph-mon[29101]: pgmap v758: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:28:51.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:51 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:28:51.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:51 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:28:51.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:51 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-94"}]: dispatch 2024-03-26T04:28:51.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:51 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-94"}]: dispatch 2024-03-26T04:28:51.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:51 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:28:51.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:51 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:28:51.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:51 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-94"}]: dispatch 2024-03-26T04:28:51.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:51 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-94"}]: dispatch 2024-03-26T04:28:51.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:51 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:28:51.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:51 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:28:51.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:51 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-94"}]: dispatch 2024-03-26T04:28:51.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:51 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-94"}]: dispatch 2024-03-26T04:28:52.042 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:28:51 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=cleanup t=2024-03-26T04:28:51.765902108Z level=info msg="Completed cleanup jobs" duration=2.345342ms 2024-03-26T04:28:52.042 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:28:51 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=sqlstore.transactions t=2024-03-26T04:28:51.774496865Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-26T04:28:52.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:52 smithi183 ceph-mon[33745]: pgmap v759: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:28:52.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:52 smithi183 ceph-mon[33745]: osdmap e497: 8 total, 8 up, 8 in 2024-03-26T04:28:52.542 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:28:52 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:28:52.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:52 smithi016 ceph-mon[37215]: pgmap v759: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:28:52.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:52 smithi016 ceph-mon[37215]: osdmap e497: 8 total, 8 up, 8 in 2024-03-26T04:28:52.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:52 smithi016 ceph-mon[29101]: pgmap v759: 292 pgs: 292 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:28:52.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:52 smithi016 ceph-mon[29101]: osdmap e497: 8 total, 8 up, 8 in 2024-03-26T04:28:53.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:53 smithi183 ceph-mon[33745]: osdmap e498: 8 total, 8 up, 8 in 2024-03-26T04:28:53.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:53 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:28:53.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:53 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:28:53.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:53 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:28:53.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:53 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:28:53.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:53 smithi016 ceph-mon[37215]: osdmap e498: 8 total, 8 up, 8 in 2024-03-26T04:28:53.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:53 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:28:53.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:53 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:28:53.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:53 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:28:53.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:53 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:28:53.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:53 smithi016 ceph-mon[29101]: osdmap e498: 8 total, 8 up, 8 in 2024-03-26T04:28:53.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:53 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:28:53.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:53 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-96","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:28:53.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:53 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:28:53.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:53 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:28:54.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:54 smithi183 ceph-mon[33745]: pgmap v762: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:28:54.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:54 smithi183 ceph-mon[33745]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:28:54.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:54 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:28:54.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:54 smithi183 ceph-mon[33745]: osdmap e499: 8 total, 8 up, 8 in 2024-03-26T04:28:54.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:54 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:28:54.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:54 smithi016 ceph-mon[37215]: pgmap v762: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:28:54.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:54 smithi016 ceph-mon[37215]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:28:54.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:54 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:28:54.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:54 smithi016 ceph-mon[37215]: osdmap e499: 8 total, 8 up, 8 in 2024-03-26T04:28:54.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:54 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:28:54.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:54 smithi016 ceph-mon[29101]: pgmap v762: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:28:54.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:54 smithi016 ceph-mon[29101]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:28:54.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:54 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-96","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:28:54.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:54 smithi016 ceph-mon[29101]: osdmap e499: 8 total, 8 up, 8 in 2024-03-26T04:28:54.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:54 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:28:55.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:55 smithi016 ceph-mon[37215]: osdmap e500: 8 total, 8 up, 8 in 2024-03-26T04:28:55.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:55 smithi016 ceph-mon[29101]: osdmap e500: 8 total, 8 up, 8 in 2024-03-26T04:28:55.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:55 smithi183 ceph-mon[33745]: osdmap e500: 8 total, 8 up, 8 in 2024-03-26T04:28:56.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:56 smithi016 ceph-mon[29101]: pgmap v765: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:28:56.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:56 smithi016 ceph-mon[37215]: pgmap v765: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:28:56.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:56 smithi183 ceph-mon[33745]: pgmap v765: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:28:58.594 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:28:58 smithi183 ceph-mon[33745]: pgmap v766: 292 pgs: 12 unknown, 280 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 347 B/s wr, 2 op/s 2024-03-26T04:28:58.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:28:58 smithi016 ceph-mon[37215]: pgmap v766: 292 pgs: 12 unknown, 280 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 347 B/s wr, 2 op/s 2024-03-26T04:28:58.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:28:58 smithi016 ceph-mon[29101]: pgmap v766: 292 pgs: 12 unknown, 280 active+clean; 8.4 MiB data, 639 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 347 B/s wr, 2 op/s 2024-03-26T04:28:59.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:28:58 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:28:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:29:00.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:00 smithi016 ceph-mon[37215]: pgmap v767: 292 pgs: 292 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 742 B/s rd, 742 B/s wr, 2 op/s 2024-03-26T04:29:00.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:00 smithi016 ceph-mon[29101]: pgmap v767: 292 pgs: 292 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 742 B/s rd, 742 B/s wr, 2 op/s 2024-03-26T04:29:00.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:00 smithi183 ceph-mon[33745]: pgmap v767: 292 pgs: 292 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 742 B/s rd, 742 B/s wr, 2 op/s 2024-03-26T04:29:01.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:01 smithi016 ceph-mon[37215]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:29:01.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:01 smithi016 ceph-mon[29101]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:29:01.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:01 smithi183 ceph-mon[33745]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:29:02.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:02 smithi016 ceph-mon[37215]: pgmap v768: 292 pgs: 292 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 639 B/s wr, 1 op/s 2024-03-26T04:29:02.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:02 smithi016 ceph-mon[29101]: pgmap v768: 292 pgs: 292 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 639 B/s wr, 1 op/s 2024-03-26T04:29:02.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:02 smithi183 ceph-mon[33745]: pgmap v768: 292 pgs: 292 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 639 B/s wr, 1 op/s 2024-03-26T04:29:02.791 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:29:02 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:29:03.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:03 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:29:03.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:03 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:29:03.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:03 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:29:04.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:04 smithi016 ceph-mon[29101]: pgmap v769: 292 pgs: 292 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 517 B/s wr, 2 op/s 2024-03-26T04:29:04.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:04 smithi016 ceph-mon[37215]: pgmap v769: 292 pgs: 292 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 517 B/s wr, 2 op/s 2024-03-26T04:29:04.663 INFO:journalctl@ceph.osd.1.smithi016.stdout:Mar 26 04:29:04 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-1[46918]: 2024-03-26T04:29:04.316+0000 7ff47fd0a700 -1 snap_mapper.add_oid found existing snaps mapped on 110:fb160d74:test-rados-api-smithi016-74534-97::foo:21, removing 2024-03-26T04:29:04.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:04 smithi183 ceph-mon[33745]: pgmap v769: 292 pgs: 292 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 517 B/s wr, 2 op/s 2024-03-26T04:29:04.792 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 26 04:29:04 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-5[44571]: 2024-03-26T04:29:04.317+0000 7fcfee339700 -1 snap_mapper.add_oid found existing snaps mapped on 110:fb160d74:test-rados-api-smithi016-74534-97::foo:21, removing 2024-03-26T04:29:04.792 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:29:04 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:29:04.317+0000 7f37d7ebe700 -1 snap_mapper.add_oid found existing snaps mapped on 110:fb160d74:test-rados-api-smithi016-74534-97::foo:21, removing 2024-03-26T04:29:05.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:05 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:29:05.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:05 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:29:05.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:05 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-96"}]: dispatch 2024-03-26T04:29:05.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:05 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-96"}]: dispatch 2024-03-26T04:29:05.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:05 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:29:05.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:05 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:29:05.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:05 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-96"}]: dispatch 2024-03-26T04:29:05.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:05 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-96"}]: dispatch 2024-03-26T04:29:05.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:05 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:29:05.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:05 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:29:05.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:05 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-96"}]: dispatch 2024-03-26T04:29:05.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:05 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-96"}]: dispatch 2024-03-26T04:29:06.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:06 smithi016 ceph-mon[37215]: pgmap v770: 292 pgs: 292 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 940 B/s rd, 470 B/s wr, 1 op/s 2024-03-26T04:29:06.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:06 smithi016 ceph-mon[37215]: osdmap e501: 8 total, 8 up, 8 in 2024-03-26T04:29:06.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:06 smithi016 ceph-mon[29101]: pgmap v770: 292 pgs: 292 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 940 B/s rd, 470 B/s wr, 1 op/s 2024-03-26T04:29:06.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:06 smithi016 ceph-mon[29101]: osdmap e501: 8 total, 8 up, 8 in 2024-03-26T04:29:06.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:06 smithi183 ceph-mon[33745]: pgmap v770: 292 pgs: 292 active+clean; 8.4 MiB data, 644 MiB used, 715 GiB / 715 GiB avail; 940 B/s rd, 470 B/s wr, 1 op/s 2024-03-26T04:29:06.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:06 smithi183 ceph-mon[33745]: osdmap e501: 8 total, 8 up, 8 in 2024-03-26T04:29:07.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:07 smithi016 ceph-mon[37215]: osdmap e502: 8 total, 8 up, 8 in 2024-03-26T04:29:07.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:07 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:29:07.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:07 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:29:07.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:07 smithi016 ceph-mon[29101]: osdmap e502: 8 total, 8 up, 8 in 2024-03-26T04:29:07.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:07 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:29:07.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:07 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:29:07.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:07 smithi183 ceph-mon[33745]: osdmap e502: 8 total, 8 up, 8 in 2024-03-26T04:29:07.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:07 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:29:07.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:07 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-98","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:29:08.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:08 smithi016 ceph-mon[37215]: pgmap v773: 292 pgs: 21 creating+peering, 11 unknown, 260 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T04:29:08.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:08 smithi016 ceph-mon[37215]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:29:08.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:08 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:29:08.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:08 smithi016 ceph-mon[37215]: osdmap e503: 8 total, 8 up, 8 in 2024-03-26T04:29:08.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:08 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:29:08.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:08 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:29:08.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:08 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:29:08.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:08 smithi016 ceph-mon[29101]: pgmap v773: 292 pgs: 21 creating+peering, 11 unknown, 260 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T04:29:08.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:08 smithi016 ceph-mon[29101]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:29:08.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:08 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:29:08.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:08 smithi016 ceph-mon[29101]: osdmap e503: 8 total, 8 up, 8 in 2024-03-26T04:29:08.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:08 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:29:08.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:08 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:29:08.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:08 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:29:08.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:08 smithi183 ceph-mon[33745]: pgmap v773: 292 pgs: 21 creating+peering, 11 unknown, 260 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T04:29:08.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:08 smithi183 ceph-mon[33745]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:29:08.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:08 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-98","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:29:08.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:08 smithi183 ceph-mon[33745]: osdmap e503: 8 total, 8 up, 8 in 2024-03-26T04:29:08.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:08 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:29:08.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:08 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-98", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:29:08.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:08 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:29:09.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:29:08 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:29:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:29:09.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:09 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-98", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:29:09.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:09 smithi016 ceph-mon[29101]: osdmap e504: 8 total, 8 up, 8 in 2024-03-26T04:29:09.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:09 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-98", "mode": "writeback"}]: dispatch 2024-03-26T04:29:09.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:09 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-98", "mode": "writeback"}]: dispatch 2024-03-26T04:29:09.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:09 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-98", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:29:09.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:09 smithi016 ceph-mon[37215]: osdmap e504: 8 total, 8 up, 8 in 2024-03-26T04:29:09.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:09 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-98", "mode": "writeback"}]: dispatch 2024-03-26T04:29:09.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:09 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-98", "mode": "writeback"}]: dispatch 2024-03-26T04:29:09.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:09 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-98", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:29:09.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:09 smithi183 ceph-mon[33745]: osdmap e504: 8 total, 8 up, 8 in 2024-03-26T04:29:09.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:09 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-98", "mode": "writeback"}]: dispatch 2024-03-26T04:29:09.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:09 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-98", "mode": "writeback"}]: dispatch 2024-03-26T04:29:10.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:10 smithi016 ceph-mon[29101]: pgmap v776: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 op/s 2024-03-26T04:29:10.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:10 smithi016 ceph-mon[29101]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:29:10.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:10 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-98", "mode": "writeback"}]': finished 2024-03-26T04:29:10.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:10 smithi016 ceph-mon[29101]: osdmap e505: 8 total, 8 up, 8 in 2024-03-26T04:29:10.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:10 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-98"}]: dispatch 2024-03-26T04:29:10.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:10 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-98"}]: dispatch 2024-03-26T04:29:10.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:10 smithi016 ceph-mon[37215]: pgmap v776: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 op/s 2024-03-26T04:29:10.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:10 smithi016 ceph-mon[37215]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:29:10.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:10 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-98", "mode": "writeback"}]': finished 2024-03-26T04:29:10.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:10 smithi016 ceph-mon[37215]: osdmap e505: 8 total, 8 up, 8 in 2024-03-26T04:29:10.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:10 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-98"}]: dispatch 2024-03-26T04:29:10.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:10 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-98"}]: dispatch 2024-03-26T04:29:10.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:10 smithi183 ceph-mon[33745]: pgmap v776: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 op/s 2024-03-26T04:29:10.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:10 smithi183 ceph-mon[33745]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:29:10.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:10 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-98", "mode": "writeback"}]': finished 2024-03-26T04:29:10.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:10 smithi183 ceph-mon[33745]: osdmap e505: 8 total, 8 up, 8 in 2024-03-26T04:29:10.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:10 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-98"}]: dispatch 2024-03-26T04:29:10.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:10 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-98"}]: dispatch 2024-03-26T04:29:11.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:11 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-98"}]': finished 2024-03-26T04:29:11.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:11 smithi016 ceph-mon[29101]: osdmap e506: 8 total, 8 up, 8 in 2024-03-26T04:29:11.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:11 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:29:11.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:11 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:29:11.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:11 smithi016 ceph-mon[29101]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:29:11.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:11 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-98","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T04:29:11.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:11 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-98"}]': finished 2024-03-26T04:29:11.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:11 smithi016 ceph-mon[37215]: osdmap e506: 8 total, 8 up, 8 in 2024-03-26T04:29:11.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:11 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:29:11.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:11 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:29:11.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:11 smithi016 ceph-mon[37215]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:29:11.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:11 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-98","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T04:29:11.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:11 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-98"}]': finished 2024-03-26T04:29:11.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:11 smithi183 ceph-mon[33745]: osdmap e506: 8 total, 8 up, 8 in 2024-03-26T04:29:11.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:11 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:29:11.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:11 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-98","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:29:11.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:11 smithi183 ceph-mon[33745]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:29:11.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:11 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-98","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T04:29:12.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:12 smithi016 ceph-mon[29101]: pgmap v779: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 255 B/s wr, 0 op/s 2024-03-26T04:29:12.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:12 smithi016 ceph-mon[29101]: osdmap e507: 8 total, 8 up, 8 in 2024-03-26T04:29:12.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:12 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-26T04:29:12.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:12 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-26T04:29:12.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:12 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-98","var": "hit_set_count","val": "1"}]': finished 2024-03-26T04:29:12.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:12 smithi016 ceph-mon[37215]: pgmap v779: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 255 B/s wr, 0 op/s 2024-03-26T04:29:12.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:12 smithi016 ceph-mon[37215]: osdmap e507: 8 total, 8 up, 8 in 2024-03-26T04:29:12.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:12 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-26T04:29:12.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:12 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-26T04:29:12.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:12 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-98","var": "hit_set_count","val": "1"}]': finished 2024-03-26T04:29:12.791 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:29:12 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:29:12.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:12 smithi183 ceph-mon[33745]: pgmap v779: 292 pgs: 32 creating+peering, 260 active+clean; 8.4 MiB data, 648 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 255 B/s wr, 0 op/s 2024-03-26T04:29:12.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:12 smithi183 ceph-mon[33745]: osdmap e507: 8 total, 8 up, 8 in 2024-03-26T04:29:12.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:12 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-26T04:29:12.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:12 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-98","var": "hit_set_count","val": "1"}]: dispatch 2024-03-26T04:29:12.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:12 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-98","var": "hit_set_count","val": "1"}]': finished 2024-03-26T04:29:13.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:13 smithi016 ceph-mon[37215]: osdmap e508: 8 total, 8 up, 8 in 2024-03-26T04:29:13.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:13 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:29:13.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:13 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:29:13.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:13 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:29:13.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:13 smithi016 ceph-mon[29101]: osdmap e508: 8 total, 8 up, 8 in 2024-03-26T04:29:13.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:13 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:29:13.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:13 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:29:13.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:13 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:29:13.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:13 smithi183 ceph-mon[33745]: osdmap e508: 8 total, 8 up, 8 in 2024-03-26T04:29:13.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:13 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:29:13.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:13 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-98","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:29:13.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:13 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:29:14.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:14 smithi016 ceph-mon[29101]: pgmap v782: 292 pgs: 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:29:14.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:14 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-98","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:29:14.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:14 smithi016 ceph-mon[29101]: osdmap e509: 8 total, 8 up, 8 in 2024-03-26T04:29:14.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:14 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-26T04:29:14.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:14 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-26T04:29:14.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:14 smithi016 ceph-mon[37215]: pgmap v782: 292 pgs: 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:29:14.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:14 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-98","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:29:14.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:14 smithi016 ceph-mon[37215]: osdmap e509: 8 total, 8 up, 8 in 2024-03-26T04:29:14.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:14 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-26T04:29:14.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:14 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-26T04:29:14.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:14 smithi183 ceph-mon[33745]: pgmap v782: 292 pgs: 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:29:14.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:14 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-98","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:29:14.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:14 smithi183 ceph-mon[33745]: osdmap e509: 8 total, 8 up, 8 in 2024-03-26T04:29:14.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:14 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-26T04:29:14.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:14 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-98","var": "target_max_objects","val": "250"}]: dispatch 2024-03-26T04:29:15.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:15 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-98","var": "target_max_objects","val": "250"}]': finished 2024-03-26T04:29:15.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:15 smithi016 ceph-mon[29101]: osdmap e510: 8 total, 8 up, 8 in 2024-03-26T04:29:15.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:15 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:29:15.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:15 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:29:15.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:15 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-98","var": "target_max_objects","val": "250"}]': finished 2024-03-26T04:29:15.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:15 smithi016 ceph-mon[37215]: osdmap e510: 8 total, 8 up, 8 in 2024-03-26T04:29:15.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:15 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:29:15.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:15 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:29:15.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:15 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-98","var": "target_max_objects","val": "250"}]': finished 2024-03-26T04:29:15.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:15 smithi183 ceph-mon[33745]: osdmap e510: 8 total, 8 up, 8 in 2024-03-26T04:29:15.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:15 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:29:15.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:15 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:29:16.388 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:16 smithi016 ceph-mon[29101]: pgmap v785: 292 pgs: 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:29:16.388 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:16 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:29:16.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:16 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-98"}]: dispatch 2024-03-26T04:29:16.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:16 smithi016 ceph-mon[29101]: osdmap e511: 8 total, 8 up, 8 in 2024-03-26T04:29:16.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:16 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-98"}]: dispatch 2024-03-26T04:29:16.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:16 smithi016 ceph-mon[29101]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:29:16.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:16 smithi016 ceph-mon[37215]: pgmap v785: 292 pgs: 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:29:16.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:16 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:29:16.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:16 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-98"}]: dispatch 2024-03-26T04:29:16.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:16 smithi016 ceph-mon[37215]: osdmap e511: 8 total, 8 up, 8 in 2024-03-26T04:29:16.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:16 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-98"}]: dispatch 2024-03-26T04:29:16.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:16 smithi016 ceph-mon[37215]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:29:16.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:16 smithi183 ceph-mon[33745]: pgmap v785: 292 pgs: 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:29:16.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:16 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:29:16.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:16 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-98"}]: dispatch 2024-03-26T04:29:16.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:16 smithi183 ceph-mon[33745]: osdmap e511: 8 total, 8 up, 8 in 2024-03-26T04:29:16.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:16 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-98"}]: dispatch 2024-03-26T04:29:16.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:16 smithi183 ceph-mon[33745]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:29:17.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:17 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-98"}]': finished 2024-03-26T04:29:17.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:17 smithi016 ceph-mon[29101]: osdmap e512: 8 total, 8 up, 8 in 2024-03-26T04:29:17.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:17 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-98"}]': finished 2024-03-26T04:29:17.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:17 smithi016 ceph-mon[37215]: osdmap e512: 8 total, 8 up, 8 in 2024-03-26T04:29:17.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:17 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-98"}]': finished 2024-03-26T04:29:17.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:17 smithi183 ceph-mon[33745]: osdmap e512: 8 total, 8 up, 8 in 2024-03-26T04:29:18.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:18 smithi016 ceph-mon[29101]: pgmap v788: 292 pgs: 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:29:18.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:18 smithi016 ceph-mon[29101]: osdmap e513: 8 total, 8 up, 8 in 2024-03-26T04:29:18.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:18 smithi016 ceph-mon[37215]: pgmap v788: 292 pgs: 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:29:18.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:18 smithi016 ceph-mon[37215]: osdmap e513: 8 total, 8 up, 8 in 2024-03-26T04:29:18.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:18 smithi183 ceph-mon[33745]: pgmap v788: 292 pgs: 292 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:29:18.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:18 smithi183 ceph-mon[33745]: osdmap e513: 8 total, 8 up, 8 in 2024-03-26T04:29:19.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:29:18 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:29:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:29:19.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:19 smithi016 ceph-mon[29101]: osdmap e514: 8 total, 8 up, 8 in 2024-03-26T04:29:19.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:19 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:29:19.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:19 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:29:19.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:19 smithi016 ceph-mon[37215]: osdmap e514: 8 total, 8 up, 8 in 2024-03-26T04:29:19.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:19 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:29:19.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:19 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:29:19.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:19 smithi183 ceph-mon[33745]: osdmap e514: 8 total, 8 up, 8 in 2024-03-26T04:29:19.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:19 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:29:19.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:19 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-100","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:29:20.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:20 smithi183 ceph-mon[33745]: pgmap v791: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:29:20.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:20 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:29:20.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:20 smithi183 ceph-mon[33745]: osdmap e515: 8 total, 8 up, 8 in 2024-03-26T04:29:20.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:20 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:29:20.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:20 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:29:20.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:20 smithi016 ceph-mon[29101]: pgmap v791: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:29:20.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:20 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:29:20.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:20 smithi016 ceph-mon[29101]: osdmap e515: 8 total, 8 up, 8 in 2024-03-26T04:29:20.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:20 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:29:20.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:20 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:29:20.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:20 smithi016 ceph-mon[37215]: pgmap v791: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:29:20.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:20 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-100","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:29:20.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:20 smithi016 ceph-mon[37215]: osdmap e515: 8 total, 8 up, 8 in 2024-03-26T04:29:20.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:20 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:29:20.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:20 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-100", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:29:21.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:21 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-100", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:29:21.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:21 smithi183 ceph-mon[33745]: osdmap e516: 8 total, 8 up, 8 in 2024-03-26T04:29:21.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:21 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-100"}]: dispatch 2024-03-26T04:29:21.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:21 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-100"}]: dispatch 2024-03-26T04:29:21.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:21 smithi183 ceph-mon[33745]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:29:21.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:21 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-100"}]': finished 2024-03-26T04:29:21.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:21 smithi183 ceph-mon[33745]: osdmap e517: 8 total, 8 up, 8 in 2024-03-26T04:29:21.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:21 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-100", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:29:21.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:21 smithi016 ceph-mon[29101]: osdmap e516: 8 total, 8 up, 8 in 2024-03-26T04:29:21.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:21 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-100"}]: dispatch 2024-03-26T04:29:21.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:21 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-100"}]: dispatch 2024-03-26T04:29:21.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:21 smithi016 ceph-mon[29101]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:29:21.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:21 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-100"}]': finished 2024-03-26T04:29:21.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:21 smithi016 ceph-mon[29101]: osdmap e517: 8 total, 8 up, 8 in 2024-03-26T04:29:21.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:21 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-100", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:29:21.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:21 smithi016 ceph-mon[37215]: osdmap e516: 8 total, 8 up, 8 in 2024-03-26T04:29:21.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:21 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-100"}]: dispatch 2024-03-26T04:29:21.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:21 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-100"}]: dispatch 2024-03-26T04:29:21.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:21 smithi016 ceph-mon[37215]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:29:21.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:21 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-6", "overlaypool": "test-rados-api-smithi016-74534-100"}]': finished 2024-03-26T04:29:21.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:21 smithi016 ceph-mon[37215]: osdmap e517: 8 total, 8 up, 8 in 2024-03-26T04:29:22.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:22 smithi183 ceph-mon[33745]: pgmap v794: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:29:22.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:22 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-100", "mode": "writeback"}]: dispatch 2024-03-26T04:29:22.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:22 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-100", "mode": "writeback"}]: dispatch 2024-03-26T04:29:22.792 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:29:22 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:29:22.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:22 smithi016 ceph-mon[37215]: pgmap v794: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:29:22.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:22 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-100", "mode": "writeback"}]: dispatch 2024-03-26T04:29:22.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:22 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-100", "mode": "writeback"}]: dispatch 2024-03-26T04:29:22.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:22 smithi016 ceph-mon[29101]: pgmap v794: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 649 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:29:22.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:22 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-100", "mode": "writeback"}]: dispatch 2024-03-26T04:29:22.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:22 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-100", "mode": "writeback"}]: dispatch 2024-03-26T04:29:23.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:23 smithi183 ceph-mon[33745]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:29:23.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:23 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-100", "mode": "writeback"}]': finished 2024-03-26T04:29:23.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:23 smithi183 ceph-mon[33745]: osdmap e518: 8 total, 8 up, 8 in 2024-03-26T04:29:23.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:23 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:29:23.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:23 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:29:23.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:23 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:29:23.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:23 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:29:23.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:23 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:29:23.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:23 smithi016 ceph-mon[29101]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:29:23.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:23 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-100", "mode": "writeback"}]': finished 2024-03-26T04:29:23.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:23 smithi016 ceph-mon[29101]: osdmap e518: 8 total, 8 up, 8 in 2024-03-26T04:29:23.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:23 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:29:23.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:23 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:29:23.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:23 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:29:23.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:23 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:29:23.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:23 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:29:23.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:23 smithi016 ceph-mon[37215]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:29:23.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:23 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-100", "mode": "writeback"}]': finished 2024-03-26T04:29:23.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:23 smithi016 ceph-mon[37215]: osdmap e518: 8 total, 8 up, 8 in 2024-03-26T04:29:23.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:23 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:29:23.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:23 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-100","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:29:23.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:23 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:29:23.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:23 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:29:23.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:23 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:29:24.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:24 smithi183 ceph-mon[33745]: pgmap v797: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:29:24.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:24 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-100","var": "hit_set_count","val": "2"}]': finished 2024-03-26T04:29:24.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:24 smithi183 ceph-mon[33745]: osdmap e519: 8 total, 8 up, 8 in 2024-03-26T04:29:24.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:24 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:29:24.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:24 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:29:24.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:24 smithi016 ceph-mon[37215]: pgmap v797: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:29:24.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:24 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-100","var": "hit_set_count","val": "2"}]': finished 2024-03-26T04:29:24.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:24 smithi016 ceph-mon[37215]: osdmap e519: 8 total, 8 up, 8 in 2024-03-26T04:29:24.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:24 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:29:24.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:24 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:29:24.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:24 smithi016 ceph-mon[29101]: pgmap v797: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:29:24.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:24 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-100","var": "hit_set_count","val": "2"}]': finished 2024-03-26T04:29:24.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:24 smithi016 ceph-mon[29101]: osdmap e519: 8 total, 8 up, 8 in 2024-03-26T04:29:24.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:24 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:29:24.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:24 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-100","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:29:25.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:25 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-100","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:29:25.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:25 smithi183 ceph-mon[33745]: osdmap e520: 8 total, 8 up, 8 in 2024-03-26T04:29:25.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:25 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T04:29:25.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:25 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T04:29:25.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:25 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-100","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:29:25.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:25 smithi016 ceph-mon[29101]: osdmap e520: 8 total, 8 up, 8 in 2024-03-26T04:29:25.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:25 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T04:29:25.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:25 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T04:29:25.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:25 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-100","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:29:25.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:25 smithi016 ceph-mon[37215]: osdmap e520: 8 total, 8 up, 8 in 2024-03-26T04:29:25.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:25 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T04:29:25.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:25 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-100","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T04:29:26.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:26 smithi183 ceph-mon[33745]: pgmap v800: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:29:26.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:26 smithi183 ceph-mon[33745]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:29:26.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:26 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T04:29:26.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:26 smithi183 ceph-mon[33745]: osdmap e521: 8 total, 8 up, 8 in 2024-03-26T04:29:26.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:26 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-26T04:29:26.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:26 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-26T04:29:26.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:26 smithi183 ceph-mon[33745]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:29:26.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:26 smithi016 ceph-mon[29101]: pgmap v800: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:29:26.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:26 smithi016 ceph-mon[29101]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:29:26.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:26 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T04:29:26.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:26 smithi016 ceph-mon[29101]: osdmap e521: 8 total, 8 up, 8 in 2024-03-26T04:29:26.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:26 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-26T04:29:26.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:26 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-26T04:29:26.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:26 smithi016 ceph-mon[29101]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:29:26.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:26 smithi016 ceph-mon[37215]: pgmap v800: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:29:26.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:26 smithi016 ceph-mon[37215]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:29:26.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:26 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-100","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T04:29:26.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:26 smithi016 ceph-mon[37215]: osdmap e521: 8 total, 8 up, 8 in 2024-03-26T04:29:26.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:26 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-26T04:29:26.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:26 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-100","var": "min_read_recency_for_promote","val": "10000"}]: dispatch 2024-03-26T04:29:26.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:26 smithi016 ceph-mon[37215]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:29:27.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:27 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-03-26T04:29:27.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:27 smithi183 ceph-mon[33745]: osdmap e522: 8 total, 8 up, 8 in 2024-03-26T04:29:27.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:27 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:29:27.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:27 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:29:27.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:27 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-03-26T04:29:27.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:27 smithi016 ceph-mon[37215]: osdmap e522: 8 total, 8 up, 8 in 2024-03-26T04:29:27.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:27 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:29:27.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:27 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:29:27.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:27 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-100","var": "min_read_recency_for_promote","val": "10000"}]': finished 2024-03-26T04:29:27.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:27 smithi016 ceph-mon[29101]: osdmap e522: 8 total, 8 up, 8 in 2024-03-26T04:29:27.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:27 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:29:27.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:27 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:29:28.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:28 smithi016 ceph-mon[29101]: pgmap v803: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:29:28.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:28 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:29:28.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:28 smithi016 ceph-mon[29101]: osdmap e523: 8 total, 8 up, 8 in 2024-03-26T04:29:28.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:28 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-100"}]: dispatch 2024-03-26T04:29:28.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:28 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-100"}]: dispatch 2024-03-26T04:29:28.746 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:28 smithi016 ceph-mon[37215]: pgmap v803: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:29:28.747 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:28 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:29:28.747 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:28 smithi016 ceph-mon[37215]: osdmap e523: 8 total, 8 up, 8 in 2024-03-26T04:29:28.747 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:28 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-100"}]: dispatch 2024-03-26T04:29:28.747 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:28 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-100"}]: dispatch 2024-03-26T04:29:28.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:28 smithi183 ceph-mon[33745]: pgmap v803: 292 pgs: 292 active+clean; 8.4 MiB data, 654 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:29:28.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:28 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]': finished 2024-03-26T04:29:28.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:28 smithi183 ceph-mon[33745]: osdmap e523: 8 total, 8 up, 8 in 2024-03-26T04:29:28.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:28 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-100"}]: dispatch 2024-03-26T04:29:28.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:28 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-100"}]: dispatch 2024-03-26T04:29:29.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:29:28 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:29:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:29:29.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:29 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-100"}]': finished 2024-03-26T04:29:29.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:29 smithi183 ceph-mon[33745]: osdmap e524: 8 total, 8 up, 8 in 2024-03-26T04:29:29.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:29 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-100"}]': finished 2024-03-26T04:29:29.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:29 smithi016 ceph-mon[29101]: osdmap e524: 8 total, 8 up, 8 in 2024-03-26T04:29:29.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:29 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-100"}]': finished 2024-03-26T04:29:29.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:29 smithi016 ceph-mon[37215]: osdmap e524: 8 total, 8 up, 8 in 2024-03-26T04:29:30.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:30 smithi183 ceph-mon[33745]: pgmap v806: 292 pgs: 292 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T04:29:30.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:30 smithi183 ceph-mon[33745]: osdmap e525: 8 total, 8 up, 8 in 2024-03-26T04:29:30.813 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:30 smithi016 ceph-mon[29101]: pgmap v806: 292 pgs: 292 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T04:29:30.814 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:30 smithi016 ceph-mon[29101]: osdmap e525: 8 total, 8 up, 8 in 2024-03-26T04:29:30.814 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:30 smithi016 ceph-mon[37215]: pgmap v806: 292 pgs: 292 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T04:29:30.814 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:30 smithi016 ceph-mon[37215]: osdmap e525: 8 total, 8 up, 8 in 2024-03-26T04:29:31.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:31 smithi183 ceph-mon[33745]: osdmap e526: 8 total, 8 up, 8 in 2024-03-26T04:29:31.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:31 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:29:31.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:31 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:29:31.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:31 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:29:31.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:31 smithi183 ceph-mon[33745]: osdmap e527: 8 total, 8 up, 8 in 2024-03-26T04:29:31.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:31 smithi016 ceph-mon[29101]: osdmap e526: 8 total, 8 up, 8 in 2024-03-26T04:29:31.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:31 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:29:31.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:31 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:29:31.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:31 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:29:31.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:31 smithi016 ceph-mon[29101]: osdmap e527: 8 total, 8 up, 8 in 2024-03-26T04:29:31.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:31 smithi016 ceph-mon[37215]: osdmap e526: 8 total, 8 up, 8 in 2024-03-26T04:29:31.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:31 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:29:31.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:31 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-102","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:29:31.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:31 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-102","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:29:31.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:31 smithi016 ceph-mon[37215]: osdmap e527: 8 total, 8 up, 8 in 2024-03-26T04:29:32.791 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:29:32 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:29:32.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:32 smithi183 ceph-mon[33745]: pgmap v809: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2024-03-26T04:29:32.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:32 smithi183 ceph-mon[33745]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:29:32.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:32 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:29:32.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:32 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:29:32.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:32 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:29:32.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:32 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:29:32.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:32 smithi183 ceph-mon[33745]: osdmap e528: 8 total, 8 up, 8 in 2024-03-26T04:29:32.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:32 smithi016 ceph-mon[37215]: pgmap v809: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2024-03-26T04:29:32.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:32 smithi016 ceph-mon[37215]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:29:32.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:32 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:29:32.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:32 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:29:32.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:32 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:29:32.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:32 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:29:32.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:32 smithi016 ceph-mon[37215]: osdmap e528: 8 total, 8 up, 8 in 2024-03-26T04:29:32.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:32 smithi016 ceph-mon[29101]: pgmap v809: 292 pgs: 32 unknown, 260 active+clean; 8.4 MiB data, 638 MiB used, 715 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 0 op/s 2024-03-26T04:29:32.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:32 smithi016 ceph-mon[29101]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:29:32.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:32 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd dump","format":"json"}]: dispatch 2024-03-26T04:29:32.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:32 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:29:32.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:32 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-102","var": "fingerprint_algorithm","val": "sha1"}]: dispatch 2024-03-26T04:29:32.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:32 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-102","var": "fingerprint_algorithm","val": "sha1"}]': finished 2024-03-26T04:29:32.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:32 smithi016 ceph-mon[29101]: osdmap e528: 8 total, 8 up, 8 in 2024-03-26T04:29:33.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:33 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:29:33.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:33 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:29:33.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:33 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:29:33.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:33 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:29:33.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:33 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:29:33.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:33 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:29:33.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:33 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:29:33.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:33 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]: dispatch 2024-03-26T04:29:33.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:33 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:29:34.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:34 smithi183 ceph-mon[33745]: pgmap v812: 292 pgs: 292 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:29:34.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:34 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T04:29:34.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:34 smithi183 ceph-mon[33745]: osdmap e529: 8 total, 8 up, 8 in 2024-03-26T04:29:34.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:34 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:29:34.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:34 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:29:34.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:34 smithi016 ceph-mon[37215]: pgmap v812: 292 pgs: 292 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:29:34.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:34 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T04:29:34.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:34 smithi016 ceph-mon[37215]: osdmap e529: 8 total, 8 up, 8 in 2024-03-26T04:29:34.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:34 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:29:34.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:34 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:29:34.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:34 smithi016 ceph-mon[29101]: pgmap v812: 292 pgs: 292 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:29:34.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:34 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-102","var": "dedup_chunk_algorithm","val": "fastcdc"}]': finished 2024-03-26T04:29:34.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:34 smithi016 ceph-mon[29101]: osdmap e529: 8 total, 8 up, 8 in 2024-03-26T04:29:34.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:34 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:29:34.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:34 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-102","var": "dedup_cdc_chunk_size","val": "1024"}]: dispatch 2024-03-26T04:29:35.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:35 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T04:29:35.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:35 smithi183 ceph-mon[33745]: osdmap e530: 8 total, 8 up, 8 in 2024-03-26T04:29:35.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:35 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:29:35.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:35 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:29:35.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:35 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-102"}]: dispatch 2024-03-26T04:29:35.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:35 smithi183 ceph-mon[33745]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-102"}]: dispatch 2024-03-26T04:29:35.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:35 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T04:29:35.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:35 smithi016 ceph-mon[37215]: osdmap e530: 8 total, 8 up, 8 in 2024-03-26T04:29:35.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:35 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:29:35.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:35 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:29:35.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:35 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-102"}]: dispatch 2024-03-26T04:29:35.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:35 smithi016 ceph-mon[37215]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-102"}]: dispatch 2024-03-26T04:29:35.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:35 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-102","var": "dedup_cdc_chunk_size","val": "1024"}]': finished 2024-03-26T04:29:35.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:35 smithi016 ceph-mon[29101]: osdmap e530: 8 total, 8 up, 8 in 2024-03-26T04:29:35.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:35 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:29:35.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:35 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-6"}]: dispatch 2024-03-26T04:29:35.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:35 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2385926785' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-102"}]: dispatch 2024-03-26T04:29:35.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:35 smithi016 ceph-mon[29101]: from='client.31627 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-6", "tierpool": "test-rados-api-smithi016-74534-102"}]: dispatch 2024-03-26T04:29:36.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:36 smithi183 ceph-mon[33745]: pgmap v815: 292 pgs: 292 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:29:36.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:36 smithi183 ceph-mon[33745]: osdmap e531: 8 total, 8 up, 8 in 2024-03-26T04:29:36.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:36 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:29:36.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:36 smithi016 ceph-mon[37215]: pgmap v815: 292 pgs: 292 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:29:36.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:36 smithi016 ceph-mon[37215]: osdmap e531: 8 total, 8 up, 8 in 2024-03-26T04:29:36.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:36 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:29:36.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:36 smithi016 ceph-mon[29101]: pgmap v815: 292 pgs: 292 active+clean; 8.4 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:29:36.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:36 smithi016 ceph-mon[29101]: osdmap e531: 8 total, 8 up, 8 in 2024-03-26T04:29:36.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:36 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:29:37.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:37 smithi183 ceph-mon[33745]: osdmap e532: 8 total, 8 up, 8 in 2024-03-26T04:29:37.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:37 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:29:37.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:37 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:29:37.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:37 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:29:37.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:37 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/396409920' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi016-74534-104"}]: dispatch 2024-03-26T04:29:37.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:37 smithi183 ceph-mon[33745]: from='client.56693 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi016-74534-104"}]: dispatch 2024-03-26T04:29:37.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:37 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/396409920' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi016-74534-104"}]: dispatch 2024-03-26T04:29:37.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:37 smithi183 ceph-mon[33745]: from='client.56693 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi016-74534-104"}]: dispatch 2024-03-26T04:29:37.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:37 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/396409920' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi016-74534-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:29:37.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:37 smithi183 ceph-mon[33745]: from='client.56693 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi016-74534-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:29:37.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:37 smithi016 ceph-mon[29101]: osdmap e532: 8 total, 8 up, 8 in 2024-03-26T04:29:37.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:37 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:29:37.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:37 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:29:37.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:37 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:29:37.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:37 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/396409920' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi016-74534-104"}]: dispatch 2024-03-26T04:29:37.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:37 smithi016 ceph-mon[29101]: from='client.56693 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi016-74534-104"}]: dispatch 2024-03-26T04:29:37.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:37 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/396409920' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi016-74534-104"}]: dispatch 2024-03-26T04:29:37.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:37 smithi016 ceph-mon[29101]: from='client.56693 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi016-74534-104"}]: dispatch 2024-03-26T04:29:37.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:37 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/396409920' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi016-74534-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:29:37.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:37 smithi016 ceph-mon[29101]: from='client.56693 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi016-74534-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:29:37.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:37 smithi016 ceph-mon[37215]: osdmap e532: 8 total, 8 up, 8 in 2024-03-26T04:29:37.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:37 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:29:37.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:37 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:29:37.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:37 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:29:37.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:37 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/396409920' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi016-74534-104"}]: dispatch 2024-03-26T04:29:37.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:37 smithi016 ceph-mon[37215]: from='client.56693 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi016-74534-104"}]: dispatch 2024-03-26T04:29:37.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:37 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/396409920' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi016-74534-104"}]: dispatch 2024-03-26T04:29:37.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:37 smithi016 ceph-mon[37215]: from='client.56693 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi016-74534-104"}]: dispatch 2024-03-26T04:29:37.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:37 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/396409920' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi016-74534-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:29:37.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:37 smithi016 ceph-mon[37215]: from='client.56693 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi016-74534-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:29:38.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:38 smithi183 ceph-mon[33745]: pgmap v818: 228 pgs: 228 active+clean; 582 KiB data, 639 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:29:38.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:38 smithi183 ceph-mon[33745]: from='client.56693 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi016-74534-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:29:38.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:38 smithi183 ceph-mon[33745]: osdmap e533: 8 total, 8 up, 8 in 2024-03-26T04:29:38.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:38 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/396409920' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi016-74534-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi016-74534-104"}]: dispatch 2024-03-26T04:29:38.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:38 smithi183 ceph-mon[33745]: from='client.56693 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi016-74534-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi016-74534-104"}]: dispatch 2024-03-26T04:29:38.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:38 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:29:38.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:38 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:29:38.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:29:38 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:29:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:29:38.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:38 smithi016 ceph-mon[37215]: pgmap v818: 228 pgs: 228 active+clean; 582 KiB data, 639 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:29:38.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:38 smithi016 ceph-mon[37215]: from='client.56693 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi016-74534-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:29:38.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:38 smithi016 ceph-mon[37215]: osdmap e533: 8 total, 8 up, 8 in 2024-03-26T04:29:38.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:38 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/396409920' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi016-74534-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi016-74534-104"}]: dispatch 2024-03-26T04:29:38.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:38 smithi016 ceph-mon[37215]: from='client.56693 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi016-74534-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi016-74534-104"}]: dispatch 2024-03-26T04:29:38.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:38 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:29:38.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:38 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:29:38.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:38 smithi016 ceph-mon[29101]: pgmap v818: 228 pgs: 228 active+clean; 582 KiB data, 639 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:29:38.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:38 smithi016 ceph-mon[29101]: from='client.56693 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosTierECPP_smithi016-74534-104", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:29:38.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:38 smithi016 ceph-mon[29101]: osdmap e533: 8 total, 8 up, 8 in 2024-03-26T04:29:38.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:38 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/396409920' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi016-74534-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi016-74534-104"}]: dispatch 2024-03-26T04:29:38.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:38 smithi016 ceph-mon[29101]: from='client.56693 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi016-74534-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi016-74534-104"}]: dispatch 2024-03-26T04:29:38.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:38 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:29:38.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:38 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:29:39.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:39 smithi183 ceph-mon[33745]: osdmap e534: 8 total, 8 up, 8 in 2024-03-26T04:29:39.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:39 smithi016 ceph-mon[29101]: osdmap e534: 8 total, 8 up, 8 in 2024-03-26T04:29:39.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:39 smithi016 ceph-mon[37215]: osdmap e534: 8 total, 8 up, 8 in 2024-03-26T04:29:40.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:40 smithi183 ceph-mon[33745]: pgmap v821: 228 pgs: 228 active+clean; 582 KiB data, 639 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:29:40.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:40 smithi183 ceph-mon[33745]: from='client.56693 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi016-74534-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi016-74534-104"}]': finished 2024-03-26T04:29:40.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:40 smithi183 ceph-mon[33745]: osdmap e535: 8 total, 8 up, 8 in 2024-03-26T04:29:40.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:40 smithi016 ceph-mon[37215]: pgmap v821: 228 pgs: 228 active+clean; 582 KiB data, 639 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:29:40.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:40 smithi016 ceph-mon[37215]: from='client.56693 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi016-74534-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi016-74534-104"}]': finished 2024-03-26T04:29:40.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:40 smithi016 ceph-mon[37215]: osdmap e535: 8 total, 8 up, 8 in 2024-03-26T04:29:40.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:40 smithi016 ceph-mon[29101]: pgmap v821: 228 pgs: 228 active+clean; 582 KiB data, 639 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:29:40.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:40 smithi016 ceph-mon[29101]: from='client.56693 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosTierECPP_smithi016-74534-104", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosTierECPP_smithi016-74534-104"}]': finished 2024-03-26T04:29:40.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:40 smithi016 ceph-mon[29101]: osdmap e535: 8 total, 8 up, 8 in 2024-03-26T04:29:41.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:41 smithi016 ceph-mon[37215]: osdmap e536: 8 total, 8 up, 8 in 2024-03-26T04:29:41.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:41 smithi016 ceph-mon[37215]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:29:41.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:41 smithi016 ceph-mon[29101]: osdmap e536: 8 total, 8 up, 8 in 2024-03-26T04:29:41.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:41 smithi016 ceph-mon[29101]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:29:42.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:41 smithi183 ceph-mon[33745]: osdmap e536: 8 total, 8 up, 8 in 2024-03-26T04:29:42.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:41 smithi183 ceph-mon[33745]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:29:42.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:42 smithi016 ceph-mon[37215]: pgmap v824: 236 pgs: 8 unknown, 228 active+clean; 582 KiB data, 639 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:29:42.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:42 smithi016 ceph-mon[37215]: osdmap e537: 8 total, 8 up, 8 in 2024-03-26T04:29:42.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:42 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/757073782' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:29:42.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:42 smithi016 ceph-mon[37215]: from='client.49222 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:29:42.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:42 smithi016 ceph-mon[29101]: pgmap v824: 236 pgs: 8 unknown, 228 active+clean; 582 KiB data, 639 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:29:42.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:42 smithi016 ceph-mon[29101]: osdmap e537: 8 total, 8 up, 8 in 2024-03-26T04:29:42.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:42 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/757073782' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:29:42.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:42 smithi016 ceph-mon[29101]: from='client.49222 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:29:43.041 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:29:42 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:29:43.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:42 smithi183 ceph-mon[33745]: pgmap v824: 236 pgs: 8 unknown, 228 active+clean; 582 KiB data, 639 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:29:43.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:42 smithi183 ceph-mon[33745]: osdmap e537: 8 total, 8 up, 8 in 2024-03-26T04:29:43.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:42 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/757073782' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:29:43.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:42 smithi183 ceph-mon[33745]: from='client.49222 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-107","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:29:43.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:43 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:29:43.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:43 smithi016 ceph-mon[37215]: from='client.49222 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:29:43.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:43 smithi016 ceph-mon[37215]: osdmap e538: 8 total, 8 up, 8 in 2024-03-26T04:29:43.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:43 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:29:43.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:43 smithi016 ceph-mon[29101]: from='client.49222 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:29:43.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:43 smithi016 ceph-mon[29101]: osdmap e538: 8 total, 8 up, 8 in 2024-03-26T04:29:44.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:43 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:29:44.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:43 smithi183 ceph-mon[33745]: from='client.49222 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-107","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:29:44.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:43 smithi183 ceph-mon[33745]: osdmap e538: 8 total, 8 up, 8 in 2024-03-26T04:29:44.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:44 smithi016 ceph-mon[37215]: pgmap v827: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:29:44.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:44 smithi016 ceph-mon[37215]: osdmap e539: 8 total, 8 up, 8 in 2024-03-26T04:29:44.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:44 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/757073782' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:29:44.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:44 smithi016 ceph-mon[37215]: from='client.49222 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:29:44.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:44 smithi016 ceph-mon[29101]: pgmap v827: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:29:44.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:44 smithi016 ceph-mon[29101]: osdmap e539: 8 total, 8 up, 8 in 2024-03-26T04:29:44.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:44 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/757073782' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:29:44.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:44 smithi016 ceph-mon[29101]: from='client.49222 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:29:45.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:44 smithi183 ceph-mon[33745]: pgmap v827: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:29:45.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:44 smithi183 ceph-mon[33745]: osdmap e539: 8 total, 8 up, 8 in 2024-03-26T04:29:45.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:44 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/757073782' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:29:45.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:44 smithi183 ceph-mon[33745]: from='client.49222 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-107-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:29:45.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:45 smithi016 ceph-mon[37215]: from='client.49222 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:29:45.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:45 smithi016 ceph-mon[37215]: osdmap e540: 8 total, 8 up, 8 in 2024-03-26T04:29:45.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:45 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/757073782' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-107", "tierpool": "test-rados-api-smithi016-74534-107-cache"}]: dispatch 2024-03-26T04:29:45.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:45 smithi016 ceph-mon[37215]: from='client.49222 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-107", "tierpool": "test-rados-api-smithi016-74534-107-cache"}]: dispatch 2024-03-26T04:29:45.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:45 smithi016 ceph-mon[29101]: from='client.49222 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:29:45.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:45 smithi016 ceph-mon[29101]: osdmap e540: 8 total, 8 up, 8 in 2024-03-26T04:29:45.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:45 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/757073782' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-107", "tierpool": "test-rados-api-smithi016-74534-107-cache"}]: dispatch 2024-03-26T04:29:45.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:45 smithi016 ceph-mon[29101]: from='client.49222 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-107", "tierpool": "test-rados-api-smithi016-74534-107-cache"}]: dispatch 2024-03-26T04:29:46.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:45 smithi183 ceph-mon[33745]: from='client.49222 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-107-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:29:46.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:45 smithi183 ceph-mon[33745]: osdmap e540: 8 total, 8 up, 8 in 2024-03-26T04:29:46.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:45 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/757073782' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-107", "tierpool": "test-rados-api-smithi016-74534-107-cache"}]: dispatch 2024-03-26T04:29:46.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:45 smithi183 ceph-mon[33745]: from='client.49222 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-107", "tierpool": "test-rados-api-smithi016-74534-107-cache"}]: dispatch 2024-03-26T04:29:46.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:46 smithi016 ceph-mon[37215]: pgmap v830: 300 pgs: 32 unknown, 32 creating+peering, 236 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:29:46.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:46 smithi016 ceph-mon[37215]: from='client.49222 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-107", "tierpool": "test-rados-api-smithi016-74534-107-cache"}]': finished 2024-03-26T04:29:46.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:46 smithi016 ceph-mon[37215]: osdmap e541: 8 total, 8 up, 8 in 2024-03-26T04:29:46.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:46 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/757073782' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-107", "overlaypool": "test-rados-api-smithi016-74534-107-cache"}]: dispatch 2024-03-26T04:29:46.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:46 smithi016 ceph-mon[37215]: from='client.49222 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-107", "overlaypool": "test-rados-api-smithi016-74534-107-cache"}]: dispatch 2024-03-26T04:29:46.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:46 smithi016 ceph-mon[37215]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:29:46.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:46 smithi016 ceph-mon[29101]: pgmap v830: 300 pgs: 32 unknown, 32 creating+peering, 236 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:29:46.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:46 smithi016 ceph-mon[29101]: from='client.49222 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-107", "tierpool": "test-rados-api-smithi016-74534-107-cache"}]': finished 2024-03-26T04:29:46.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:46 smithi016 ceph-mon[29101]: osdmap e541: 8 total, 8 up, 8 in 2024-03-26T04:29:46.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:46 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/757073782' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-107", "overlaypool": "test-rados-api-smithi016-74534-107-cache"}]: dispatch 2024-03-26T04:29:46.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:46 smithi016 ceph-mon[29101]: from='client.49222 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-107", "overlaypool": "test-rados-api-smithi016-74534-107-cache"}]: dispatch 2024-03-26T04:29:46.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:46 smithi016 ceph-mon[29101]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:29:47.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:46 smithi183 ceph-mon[33745]: pgmap v830: 300 pgs: 32 unknown, 32 creating+peering, 236 active+clean; 582 KiB data, 602 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:29:47.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:46 smithi183 ceph-mon[33745]: from='client.49222 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-107", "tierpool": "test-rados-api-smithi016-74534-107-cache"}]': finished 2024-03-26T04:29:47.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:46 smithi183 ceph-mon[33745]: osdmap e541: 8 total, 8 up, 8 in 2024-03-26T04:29:47.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:46 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/757073782' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-107", "overlaypool": "test-rados-api-smithi016-74534-107-cache"}]: dispatch 2024-03-26T04:29:47.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:46 smithi183 ceph-mon[33745]: from='client.49222 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-107", "overlaypool": "test-rados-api-smithi016-74534-107-cache"}]: dispatch 2024-03-26T04:29:47.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:46 smithi183 ceph-mon[33745]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:29:47.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:47 smithi016 ceph-mon[29101]: from='client.49222 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-107", "overlaypool": "test-rados-api-smithi016-74534-107-cache"}]': finished 2024-03-26T04:29:47.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:47 smithi016 ceph-mon[29101]: osdmap e542: 8 total, 8 up, 8 in 2024-03-26T04:29:47.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:47 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/757073782' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-107-cache", "mode": "writeback"}]: dispatch 2024-03-26T04:29:47.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:47 smithi016 ceph-mon[29101]: from='client.49222 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-107-cache", "mode": "writeback"}]: dispatch 2024-03-26T04:29:47.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:47 smithi016 ceph-mon[37215]: from='client.49222 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-107", "overlaypool": "test-rados-api-smithi016-74534-107-cache"}]': finished 2024-03-26T04:29:47.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:47 smithi016 ceph-mon[37215]: osdmap e542: 8 total, 8 up, 8 in 2024-03-26T04:29:47.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:47 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/757073782' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-107-cache", "mode": "writeback"}]: dispatch 2024-03-26T04:29:47.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:47 smithi016 ceph-mon[37215]: from='client.49222 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-107-cache", "mode": "writeback"}]: dispatch 2024-03-26T04:29:48.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:47 smithi183 ceph-mon[33745]: from='client.49222 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-107", "overlaypool": "test-rados-api-smithi016-74534-107-cache"}]': finished 2024-03-26T04:29:48.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:47 smithi183 ceph-mon[33745]: osdmap e542: 8 total, 8 up, 8 in 2024-03-26T04:29:48.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:47 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/757073782' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-107-cache", "mode": "writeback"}]: dispatch 2024-03-26T04:29:48.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:47 smithi183 ceph-mon[33745]: from='client.49222 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-107-cache", "mode": "writeback"}]: dispatch 2024-03-26T04:29:48.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:29:48 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:29:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:29:48.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:48 smithi016 ceph-mon[29101]: pgmap v833: 300 pgs: 14 unknown, 13 creating+peering, 273 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:29:48.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:48 smithi016 ceph-mon[29101]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:29:48.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:48 smithi016 ceph-mon[29101]: from='client.49222 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-107-cache", "mode": "writeback"}]': finished 2024-03-26T04:29:48.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:48 smithi016 ceph-mon[29101]: osdmap e543: 8 total, 8 up, 8 in 2024-03-26T04:29:48.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:48 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/757073782' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-107"}]: dispatch 2024-03-26T04:29:48.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:48 smithi016 ceph-mon[29101]: from='client.49222 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-107"}]: dispatch 2024-03-26T04:29:48.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:48 smithi016 ceph-mon[37215]: pgmap v833: 300 pgs: 14 unknown, 13 creating+peering, 273 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:29:48.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:48 smithi016 ceph-mon[37215]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:29:48.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:48 smithi016 ceph-mon[37215]: from='client.49222 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-107-cache", "mode": "writeback"}]': finished 2024-03-26T04:29:48.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:48 smithi016 ceph-mon[37215]: osdmap e543: 8 total, 8 up, 8 in 2024-03-26T04:29:48.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:48 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/757073782' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-107"}]: dispatch 2024-03-26T04:29:48.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:48 smithi016 ceph-mon[37215]: from='client.49222 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-107"}]: dispatch 2024-03-26T04:29:49.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:48 smithi183 ceph-mon[33745]: pgmap v833: 300 pgs: 14 unknown, 13 creating+peering, 273 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:29:49.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:48 smithi183 ceph-mon[33745]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:29:49.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:48 smithi183 ceph-mon[33745]: from='client.49222 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-107-cache", "mode": "writeback"}]': finished 2024-03-26T04:29:49.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:48 smithi183 ceph-mon[33745]: osdmap e543: 8 total, 8 up, 8 in 2024-03-26T04:29:49.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:48 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/757073782' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-107"}]: dispatch 2024-03-26T04:29:49.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:48 smithi183 ceph-mon[33745]: from='client.49222 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-107"}]: dispatch 2024-03-26T04:29:49.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:49 smithi016 ceph-mon[37215]: from='client.49222 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-107"}]': finished 2024-03-26T04:29:49.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:49 smithi016 ceph-mon[37215]: osdmap e544: 8 total, 8 up, 8 in 2024-03-26T04:29:49.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:49 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/757073782' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-107", "tierpool": "test-rados-api-smithi016-74534-107-cache"}]: dispatch 2024-03-26T04:29:49.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:49 smithi016 ceph-mon[37215]: from='client.49222 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-107", "tierpool": "test-rados-api-smithi016-74534-107-cache"}]: dispatch 2024-03-26T04:29:49.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:49 smithi016 ceph-mon[29101]: from='client.49222 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-107"}]': finished 2024-03-26T04:29:49.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:49 smithi016 ceph-mon[29101]: osdmap e544: 8 total, 8 up, 8 in 2024-03-26T04:29:49.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:49 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/757073782' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-107", "tierpool": "test-rados-api-smithi016-74534-107-cache"}]: dispatch 2024-03-26T04:29:49.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:49 smithi016 ceph-mon[29101]: from='client.49222 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-107", "tierpool": "test-rados-api-smithi016-74534-107-cache"}]: dispatch 2024-03-26T04:29:50.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:49 smithi183 ceph-mon[33745]: from='client.49222 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-107"}]': finished 2024-03-26T04:29:50.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:49 smithi183 ceph-mon[33745]: osdmap e544: 8 total, 8 up, 8 in 2024-03-26T04:29:50.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:49 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/757073782' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-107", "tierpool": "test-rados-api-smithi016-74534-107-cache"}]: dispatch 2024-03-26T04:29:50.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:49 smithi183 ceph-mon[33745]: from='client.49222 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-107", "tierpool": "test-rados-api-smithi016-74534-107-cache"}]: dispatch 2024-03-26T04:29:50.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:50 smithi016 ceph-mon[29101]: pgmap v836: 300 pgs: 300 active+clean; 582 KiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:29:50.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:50 smithi016 ceph-mon[29101]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:29:50.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:50 smithi016 ceph-mon[29101]: from='client.49222 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-107", "tierpool": "test-rados-api-smithi016-74534-107-cache"}]': finished 2024-03-26T04:29:50.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:50 smithi016 ceph-mon[29101]: osdmap e545: 8 total, 8 up, 8 in 2024-03-26T04:29:50.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:50 smithi016 ceph-mon[37215]: pgmap v836: 300 pgs: 300 active+clean; 582 KiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:29:50.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:50 smithi016 ceph-mon[37215]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:29:50.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:50 smithi016 ceph-mon[37215]: from='client.49222 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-107", "tierpool": "test-rados-api-smithi016-74534-107-cache"}]': finished 2024-03-26T04:29:50.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:50 smithi016 ceph-mon[37215]: osdmap e545: 8 total, 8 up, 8 in 2024-03-26T04:29:51.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:50 smithi183 ceph-mon[33745]: pgmap v836: 300 pgs: 300 active+clean; 582 KiB data, 607 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:29:51.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:50 smithi183 ceph-mon[33745]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:29:51.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:50 smithi183 ceph-mon[33745]: from='client.49222 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-107", "tierpool": "test-rados-api-smithi016-74534-107-cache"}]': finished 2024-03-26T04:29:51.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:50 smithi183 ceph-mon[33745]: osdmap e545: 8 total, 8 up, 8 in 2024-03-26T04:29:51.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:51 smithi016 ceph-mon[37215]: osdmap e546: 8 total, 8 up, 8 in 2024-03-26T04:29:51.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:51 smithi016 ceph-mon[29101]: osdmap e546: 8 total, 8 up, 8 in 2024-03-26T04:29:52.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:51 smithi183 ceph-mon[33745]: osdmap e546: 8 total, 8 up, 8 in 2024-03-26T04:29:52.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:52 smithi016 ceph-mon[37215]: pgmap v839: 268 pgs: 268 active+clean; 582 KiB data, 607 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:29:52.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:52 smithi016 ceph-mon[37215]: osdmap e547: 8 total, 8 up, 8 in 2024-03-26T04:29:52.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:52 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi016-74534-109"}]: dispatch 2024-03-26T04:29:52.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:52 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi016-74534-109"}]: dispatch 2024-03-26T04:29:52.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:52 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi016-74534-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:29:52.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:52 smithi016 ceph-mon[29101]: pgmap v839: 268 pgs: 268 active+clean; 582 KiB data, 607 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:29:52.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:52 smithi016 ceph-mon[29101]: osdmap e547: 8 total, 8 up, 8 in 2024-03-26T04:29:52.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:52 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi016-74534-109"}]: dispatch 2024-03-26T04:29:52.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:52 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi016-74534-109"}]: dispatch 2024-03-26T04:29:52.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:52 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi016-74534-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:29:53.041 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:29:52 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:29:53.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:52 smithi183 ceph-mon[33745]: pgmap v839: 268 pgs: 268 active+clean; 582 KiB data, 607 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:29:53.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:52 smithi183 ceph-mon[33745]: osdmap e547: 8 total, 8 up, 8 in 2024-03-26T04:29:53.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:52 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi016-74534-109"}]: dispatch 2024-03-26T04:29:53.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:52 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi016-74534-109"}]: dispatch 2024-03-26T04:29:53.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:52 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi016-74534-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:29:53.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:53 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:29:53.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:53 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi016-74534-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:29:53.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:53 smithi016 ceph-mon[37215]: osdmap e548: 8 total, 8 up, 8 in 2024-03-26T04:29:53.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:53 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi016-74534-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi016-74534-109"}]: dispatch 2024-03-26T04:29:53.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:53 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:29:53.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:53 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:29:53.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:53 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi016-74534-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:29:53.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:53 smithi016 ceph-mon[29101]: osdmap e548: 8 total, 8 up, 8 in 2024-03-26T04:29:53.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:53 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi016-74534-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi016-74534-109"}]: dispatch 2024-03-26T04:29:53.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:53 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:29:54.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:53 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:29:54.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:53 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi016-74534-109", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:29:54.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:53 smithi183 ceph-mon[33745]: osdmap e548: 8 total, 8 up, 8 in 2024-03-26T04:29:54.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:53 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi016-74534-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi016-74534-109"}]: dispatch 2024-03-26T04:29:54.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:53 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:29:54.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:54 smithi016 ceph-mon[37215]: pgmap v842: 236 pgs: 236 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:29:54.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:54 smithi016 ceph-mon[37215]: osdmap e549: 8 total, 8 up, 8 in 2024-03-26T04:29:54.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:54 smithi016 ceph-mon[29101]: pgmap v842: 236 pgs: 236 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:29:54.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:54 smithi016 ceph-mon[29101]: osdmap e549: 8 total, 8 up, 8 in 2024-03-26T04:29:55.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:54 smithi183 ceph-mon[33745]: pgmap v842: 236 pgs: 236 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:29:55.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:54 smithi183 ceph-mon[33745]: osdmap e549: 8 total, 8 up, 8 in 2024-03-26T04:29:55.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:55 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi016-74534-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi016-74534-109"}]': finished 2024-03-26T04:29:55.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:55 smithi016 ceph-mon[37215]: osdmap e550: 8 total, 8 up, 8 in 2024-03-26T04:29:55.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:55 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi016-74534-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi016-74534-109"}]': finished 2024-03-26T04:29:55.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:55 smithi016 ceph-mon[29101]: osdmap e550: 8 total, 8 up, 8 in 2024-03-26T04:29:56.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:55 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi016-74534-109", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi016-74534-109"}]': finished 2024-03-26T04:29:56.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:55 smithi183 ceph-mon[33745]: osdmap e550: 8 total, 8 up, 8 in 2024-03-26T04:29:56.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:56 smithi016 ceph-mon[29101]: pgmap v845: 244 pgs: 8 unknown, 236 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:29:56.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:56 smithi016 ceph-mon[29101]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:29:56.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:56 smithi016 ceph-mon[29101]: osdmap e551: 8 total, 8 up, 8 in 2024-03-26T04:29:56.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:56 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:29:56.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:56 smithi016 ceph-mon[37215]: pgmap v845: 244 pgs: 8 unknown, 236 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:29:56.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:56 smithi016 ceph-mon[37215]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:29:56.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:56 smithi016 ceph-mon[37215]: osdmap e551: 8 total, 8 up, 8 in 2024-03-26T04:29:56.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:56 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:29:57.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:56 smithi183 ceph-mon[33745]: pgmap v845: 244 pgs: 8 unknown, 236 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:29:57.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:56 smithi183 ceph-mon[33745]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:29:57.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:56 smithi183 ceph-mon[33745]: osdmap e551: 8 total, 8 up, 8 in 2024-03-26T04:29:57.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:56 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-109-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:29:57.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:57 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:29:57.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:57 smithi016 ceph-mon[29101]: osdmap e552: 8 total, 8 up, 8 in 2024-03-26T04:29:57.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:57 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-109", "tierpool": "test-rados-api-smithi016-74534-109-cache"}]: dispatch 2024-03-26T04:29:57.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:57 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:29:57.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:57 smithi016 ceph-mon[37215]: osdmap e552: 8 total, 8 up, 8 in 2024-03-26T04:29:57.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:57 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-109", "tierpool": "test-rados-api-smithi016-74534-109-cache"}]: dispatch 2024-03-26T04:29:58.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:57 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-109-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:29:58.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:57 smithi183 ceph-mon[33745]: osdmap e552: 8 total, 8 up, 8 in 2024-03-26T04:29:58.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:57 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-109", "tierpool": "test-rados-api-smithi016-74534-109-cache"}]: dispatch 2024-03-26T04:29:58.943 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:58 smithi183 ceph-mon[33745]: pgmap v848: 276 pgs: 15 unknown, 261 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:29:58.943 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-109", "tierpool": "test-rados-api-smithi016-74534-109-cache"}]': finished 2024-03-26T04:29:58.943 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:58 smithi183 ceph-mon[33745]: osdmap e553: 8 total, 8 up, 8 in 2024-03-26T04:29:58.943 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-109", "overlaypool": "test-rados-api-smithi016-74534-109-cache"}]: dispatch 2024-03-26T04:29:59.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:29:58 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:29:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:29:59.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:58 smithi016 ceph-mon[29101]: pgmap v848: 276 pgs: 15 unknown, 261 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:29:59.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-109", "tierpool": "test-rados-api-smithi016-74534-109-cache"}]': finished 2024-03-26T04:29:59.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:58 smithi016 ceph-mon[29101]: osdmap e553: 8 total, 8 up, 8 in 2024-03-26T04:29:59.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-109", "overlaypool": "test-rados-api-smithi016-74534-109-cache"}]: dispatch 2024-03-26T04:29:59.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:58 smithi016 ceph-mon[37215]: pgmap v848: 276 pgs: 15 unknown, 261 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:29:59.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-109", "tierpool": "test-rados-api-smithi016-74534-109-cache"}]': finished 2024-03-26T04:29:59.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:58 smithi016 ceph-mon[37215]: osdmap e553: 8 total, 8 up, 8 in 2024-03-26T04:29:59.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-109", "overlaypool": "test-rados-api-smithi016-74534-109-cache"}]: dispatch 2024-03-26T04:30:00.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:59 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-109", "overlaypool": "test-rados-api-smithi016-74534-109-cache"}]': finished 2024-03-26T04:30:00.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:59 smithi183 ceph-mon[33745]: osdmap e554: 8 total, 8 up, 8 in 2024-03-26T04:30:00.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:59 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-109-cache", "mode": "writeback"}]: dispatch 2024-03-26T04:30:00.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:59 smithi183 ceph-mon[33745]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:30:00.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:29:59 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-109-cache", "mode": "writeback"}]': finished 2024-03-26T04:30:00.063 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:59 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-109", "overlaypool": "test-rados-api-smithi016-74534-109-cache"}]': finished 2024-03-26T04:30:00.063 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:59 smithi016 ceph-mon[37215]: osdmap e554: 8 total, 8 up, 8 in 2024-03-26T04:30:00.063 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:59 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-109-cache", "mode": "writeback"}]: dispatch 2024-03-26T04:30:00.064 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:59 smithi016 ceph-mon[37215]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:30:00.064 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:29:59 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-109-cache", "mode": "writeback"}]': finished 2024-03-26T04:30:00.064 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:59 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-109", "overlaypool": "test-rados-api-smithi016-74534-109-cache"}]': finished 2024-03-26T04:30:00.064 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:59 smithi016 ceph-mon[29101]: osdmap e554: 8 total, 8 up, 8 in 2024-03-26T04:30:00.064 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:59 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-109-cache", "mode": "writeback"}]: dispatch 2024-03-26T04:30:00.064 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:59 smithi016 ceph-mon[29101]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:30:00.064 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:29:59 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-109-cache", "mode": "writeback"}]': finished 2024-03-26T04:30:01.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:00 smithi183 ceph-mon[33745]: pgmap v851: 276 pgs: 276 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:30:01.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:00 smithi183 ceph-mon[33745]: osdmap e555: 8 total, 8 up, 8 in 2024-03-26T04:30:01.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:00 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:30:01.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:00 smithi183 ceph-mon[33745]: Health detail: HEALTH_WARN 1 cache pools are missing hit_sets; 5 pool(s) do not have an application enabled 2024-03-26T04:30:01.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:00 smithi183 ceph-mon[33745]: [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets 2024-03-26T04:30:01.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:00 smithi183 ceph-mon[33745]: pool 'test-rados-api-smithi016-74534-109-cache' with cache_mode writeback needs hit_set_type to be set but it is not 2024-03-26T04:30:01.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:00 smithi183 ceph-mon[33745]: [WRN] POOL_APP_NOT_ENABLED: 5 pool(s) do not have an application enabled 2024-03-26T04:30:01.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:00 smithi183 ceph-mon[33745]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-26T04:30:01.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:00 smithi183 ceph-mon[33745]: application not enabled on pool 'ListObjectssmithi016-75481-1' 2024-03-26T04:30:01.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:00 smithi183 ceph-mon[33745]: application not enabled on pool 'Limitssmithi016-75515-1' 2024-03-26T04:30:01.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:00 smithi183 ceph-mon[33745]: application not enabled on pool 'LibRadosTierECPP_smithi016-74534-104' 2024-03-26T04:30:01.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:00 smithi183 ceph-mon[33745]: application not enabled on pool 'test-rados-api-smithi016-74534-109' 2024-03-26T04:30:01.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:00 smithi183 ceph-mon[33745]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-26T04:30:01.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:00 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-03-26T04:30:01.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:00 smithi183 ceph-mon[33745]: osdmap e556: 8 total, 8 up, 8 in 2024-03-26T04:30:01.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:00 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:30:01.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:00 smithi016 ceph-mon[37215]: pgmap v851: 276 pgs: 276 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:30:01.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:00 smithi016 ceph-mon[37215]: osdmap e555: 8 total, 8 up, 8 in 2024-03-26T04:30:01.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:00 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:30:01.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:00 smithi016 ceph-mon[37215]: Health detail: HEALTH_WARN 1 cache pools are missing hit_sets; 5 pool(s) do not have an application enabled 2024-03-26T04:30:01.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:00 smithi016 ceph-mon[37215]: [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets 2024-03-26T04:30:01.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:00 smithi016 ceph-mon[37215]: pool 'test-rados-api-smithi016-74534-109-cache' with cache_mode writeback needs hit_set_type to be set but it is not 2024-03-26T04:30:01.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:00 smithi016 ceph-mon[37215]: [WRN] POOL_APP_NOT_ENABLED: 5 pool(s) do not have an application enabled 2024-03-26T04:30:01.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:00 smithi016 ceph-mon[37215]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-26T04:30:01.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:00 smithi016 ceph-mon[37215]: application not enabled on pool 'ListObjectssmithi016-75481-1' 2024-03-26T04:30:01.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:00 smithi016 ceph-mon[37215]: application not enabled on pool 'Limitssmithi016-75515-1' 2024-03-26T04:30:01.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:00 smithi016 ceph-mon[37215]: application not enabled on pool 'LibRadosTierECPP_smithi016-74534-104' 2024-03-26T04:30:01.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:00 smithi016 ceph-mon[37215]: application not enabled on pool 'test-rados-api-smithi016-74534-109' 2024-03-26T04:30:01.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:00 smithi016 ceph-mon[37215]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-26T04:30:01.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:00 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-03-26T04:30:01.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:00 smithi016 ceph-mon[37215]: osdmap e556: 8 total, 8 up, 8 in 2024-03-26T04:30:01.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:00 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:30:01.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:00 smithi016 ceph-mon[29101]: pgmap v851: 276 pgs: 276 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:30:01.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:00 smithi016 ceph-mon[29101]: osdmap e555: 8 total, 8 up, 8 in 2024-03-26T04:30:01.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:00 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-109-cache","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:30:01.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:00 smithi016 ceph-mon[29101]: Health detail: HEALTH_WARN 1 cache pools are missing hit_sets; 5 pool(s) do not have an application enabled 2024-03-26T04:30:01.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:00 smithi016 ceph-mon[29101]: [WRN] CACHE_POOL_NO_HIT_SET: 1 cache pools are missing hit_sets 2024-03-26T04:30:01.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:00 smithi016 ceph-mon[29101]: pool 'test-rados-api-smithi016-74534-109-cache' with cache_mode writeback needs hit_set_type to be set but it is not 2024-03-26T04:30:01.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:00 smithi016 ceph-mon[29101]: [WRN] POOL_APP_NOT_ENABLED: 5 pool(s) do not have an application enabled 2024-03-26T04:30:01.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:00 smithi016 ceph-mon[29101]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-26T04:30:01.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:00 smithi016 ceph-mon[29101]: application not enabled on pool 'ListObjectssmithi016-75481-1' 2024-03-26T04:30:01.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:00 smithi016 ceph-mon[29101]: application not enabled on pool 'Limitssmithi016-75515-1' 2024-03-26T04:30:01.165 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:00 smithi016 ceph-mon[29101]: application not enabled on pool 'LibRadosTierECPP_smithi016-74534-104' 2024-03-26T04:30:01.165 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:00 smithi016 ceph-mon[29101]: application not enabled on pool 'test-rados-api-smithi016-74534-109' 2024-03-26T04:30:01.165 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:00 smithi016 ceph-mon[29101]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-26T04:30:01.165 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:00 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-109-cache","var": "hit_set_count","val": "2"}]': finished 2024-03-26T04:30:01.165 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:00 smithi016 ceph-mon[29101]: osdmap e556: 8 total, 8 up, 8 in 2024-03-26T04:30:01.165 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:00 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-109-cache","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:30:03.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:02 smithi183 ceph-mon[33745]: pgmap v854: 276 pgs: 276 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:30:03.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:02 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:30:03.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:02 smithi183 ceph-mon[33745]: osdmap e557: 8 total, 8 up, 8 in 2024-03-26T04:30:03.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:02 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T04:30:03.042 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:30:02 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:30:03.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:02 smithi016 ceph-mon[37215]: pgmap v854: 276 pgs: 276 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:30:03.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:02 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:30:03.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:02 smithi016 ceph-mon[37215]: osdmap e557: 8 total, 8 up, 8 in 2024-03-26T04:30:03.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:02 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T04:30:03.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:02 smithi016 ceph-mon[29101]: pgmap v854: 276 pgs: 276 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:30:03.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:02 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-109-cache","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:30:03.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:02 smithi016 ceph-mon[29101]: osdmap e557: 8 total, 8 up, 8 in 2024-03-26T04:30:03.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:02 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-109-cache","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T04:30:04.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:03 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:30:04.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:03 smithi183 ceph-mon[33745]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:30:04.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:03 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T04:30:04.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:03 smithi183 ceph-mon[33745]: osdmap e558: 8 total, 8 up, 8 in 2024-03-26T04:30:04.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:03 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-26T04:30:04.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:03 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-03-26T04:30:04.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:03 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:30:04.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:03 smithi016 ceph-mon[29101]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:30:04.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:03 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T04:30:04.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:03 smithi016 ceph-mon[29101]: osdmap e558: 8 total, 8 up, 8 in 2024-03-26T04:30:04.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:03 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-26T04:30:04.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:03 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-03-26T04:30:04.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:03 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:30:04.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:03 smithi016 ceph-mon[37215]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:30:04.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:03 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-109-cache","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T04:30:04.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:03 smithi016 ceph-mon[37215]: osdmap e558: 8 total, 8 up, 8 in 2024-03-26T04:30:04.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:03 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-109-cache","var": "min_read_recency_for_promote","val": "4"}]: dispatch 2024-03-26T04:30:04.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:03 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-109-cache","var": "min_read_recency_for_promote","val": "4"}]': finished 2024-03-26T04:30:05.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:04 smithi183 ceph-mon[33745]: pgmap v857: 276 pgs: 276 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:30:05.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:04 smithi183 ceph-mon[33745]: osdmap e559: 8 total, 8 up, 8 in 2024-03-26T04:30:05.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:04 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-109"}]: dispatch 2024-03-26T04:30:05.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:04 smithi016 ceph-mon[37215]: pgmap v857: 276 pgs: 276 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:30:05.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:04 smithi016 ceph-mon[37215]: osdmap e559: 8 total, 8 up, 8 in 2024-03-26T04:30:05.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:04 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-109"}]: dispatch 2024-03-26T04:30:05.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:04 smithi016 ceph-mon[29101]: pgmap v857: 276 pgs: 276 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:30:05.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:04 smithi016 ceph-mon[29101]: osdmap e559: 8 total, 8 up, 8 in 2024-03-26T04:30:05.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:04 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-109"}]: dispatch 2024-03-26T04:30:06.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:05 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-109"}]': finished 2024-03-26T04:30:06.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:05 smithi183 ceph-mon[33745]: osdmap e560: 8 total, 8 up, 8 in 2024-03-26T04:30:06.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:05 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-109", "tierpool": "test-rados-api-smithi016-74534-109-cache"}]: dispatch 2024-03-26T04:30:06.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:05 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-109"}]': finished 2024-03-26T04:30:06.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:05 smithi016 ceph-mon[37215]: osdmap e560: 8 total, 8 up, 8 in 2024-03-26T04:30:06.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:05 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-109", "tierpool": "test-rados-api-smithi016-74534-109-cache"}]: dispatch 2024-03-26T04:30:06.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:05 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-109"}]': finished 2024-03-26T04:30:06.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:05 smithi016 ceph-mon[29101]: osdmap e560: 8 total, 8 up, 8 in 2024-03-26T04:30:06.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:05 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-109", "tierpool": "test-rados-api-smithi016-74534-109-cache"}]: dispatch 2024-03-26T04:30:07.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:06 smithi183 ceph-mon[33745]: pgmap v860: 276 pgs: 276 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:30:07.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:06 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-109", "tierpool": "test-rados-api-smithi016-74534-109-cache"}]': finished 2024-03-26T04:30:07.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:06 smithi183 ceph-mon[33745]: osdmap e561: 8 total, 8 up, 8 in 2024-03-26T04:30:07.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:06 smithi016 ceph-mon[37215]: pgmap v860: 276 pgs: 276 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:30:07.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:06 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-109", "tierpool": "test-rados-api-smithi016-74534-109-cache"}]': finished 2024-03-26T04:30:07.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:06 smithi016 ceph-mon[37215]: osdmap e561: 8 total, 8 up, 8 in 2024-03-26T04:30:07.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:06 smithi016 ceph-mon[29101]: pgmap v860: 276 pgs: 276 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:30:07.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:06 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-109", "tierpool": "test-rados-api-smithi016-74534-109-cache"}]': finished 2024-03-26T04:30:07.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:06 smithi016 ceph-mon[29101]: osdmap e561: 8 total, 8 up, 8 in 2024-03-26T04:30:08.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:07 smithi183 ceph-mon[33745]: osdmap e562: 8 total, 8 up, 8 in 2024-03-26T04:30:08.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:07 smithi183 ceph-mon[33745]: osdmap e563: 8 total, 8 up, 8 in 2024-03-26T04:30:08.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:07 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi016-74534-109"}]: dispatch 2024-03-26T04:30:08.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:07 smithi016 ceph-mon[37215]: osdmap e562: 8 total, 8 up, 8 in 2024-03-26T04:30:08.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:07 smithi016 ceph-mon[37215]: osdmap e563: 8 total, 8 up, 8 in 2024-03-26T04:30:08.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:07 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi016-74534-109"}]: dispatch 2024-03-26T04:30:08.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:07 smithi016 ceph-mon[29101]: osdmap e562: 8 total, 8 up, 8 in 2024-03-26T04:30:08.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:07 smithi016 ceph-mon[29101]: osdmap e563: 8 total, 8 up, 8 in 2024-03-26T04:30:08.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:07 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi016-74534-109"}]: dispatch 2024-03-26T04:30:09.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:08 smithi183 ceph-mon[33745]: pgmap v863: 244 pgs: 244 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:30:09.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:08 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:30:09.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:08 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi016-74534-109"}]': finished 2024-03-26T04:30:09.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:08 smithi183 ceph-mon[33745]: osdmap e564: 8 total, 8 up, 8 in 2024-03-26T04:30:09.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:08 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi016-74534-109"}]: dispatch 2024-03-26T04:30:09.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:08 smithi016 ceph-mon[29101]: pgmap v863: 244 pgs: 244 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:30:09.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:08 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:30:09.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:08 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi016-74534-109"}]': finished 2024-03-26T04:30:09.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:08 smithi016 ceph-mon[29101]: osdmap e564: 8 total, 8 up, 8 in 2024-03-26T04:30:09.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:08 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi016-74534-109"}]: dispatch 2024-03-26T04:30:09.163 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:30:08 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:30:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:30:09.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:08 smithi016 ceph-mon[37215]: pgmap v863: 244 pgs: 244 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:30:09.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:08 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:30:09.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:08 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi016-74534-109"}]': finished 2024-03-26T04:30:09.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:08 smithi016 ceph-mon[37215]: osdmap e564: 8 total, 8 up, 8 in 2024-03-26T04:30:09.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:08 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi016-74534-109"}]: dispatch 2024-03-26T04:30:11.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:10 smithi183 ceph-mon[33745]: pgmap v866: 236 pgs: 236 active+clean; 582 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:30:11.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:10 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi016-74534-109"}]': finished 2024-03-26T04:30:11.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:10 smithi183 ceph-mon[33745]: osdmap e565: 8 total, 8 up, 8 in 2024-03-26T04:30:11.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:10 smithi183 ceph-mon[33745]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:30:11.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:10 smithi016 ceph-mon[29101]: pgmap v866: 236 pgs: 236 active+clean; 582 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:30:11.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:10 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi016-74534-109"}]': finished 2024-03-26T04:30:11.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:10 smithi016 ceph-mon[29101]: osdmap e565: 8 total, 8 up, 8 in 2024-03-26T04:30:11.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:10 smithi016 ceph-mon[29101]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:30:11.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:10 smithi016 ceph-mon[37215]: pgmap v866: 236 pgs: 236 active+clean; 582 KiB data, 593 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:30:11.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:10 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/700975982' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi016-74534-109"}]': finished 2024-03-26T04:30:11.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:10 smithi016 ceph-mon[37215]: osdmap e565: 8 total, 8 up, 8 in 2024-03-26T04:30:11.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:10 smithi016 ceph-mon[37215]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:30:12.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:11 smithi183 ceph-mon[33745]: osdmap e566: 8 total, 8 up, 8 in 2024-03-26T04:30:12.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:11 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/396409920' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi016-74534-104"}]: dispatch 2024-03-26T04:30:12.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:11 smithi183 ceph-mon[33745]: from='client.56693 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi016-74534-104"}]: dispatch 2024-03-26T04:30:12.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:11 smithi183 ceph-mon[33745]: pgmap v869: 228 pgs: 228 active+clean; 582 KiB data, 593 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:30:12.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:11 smithi016 ceph-mon[29101]: osdmap e566: 8 total, 8 up, 8 in 2024-03-26T04:30:12.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:11 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/396409920' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi016-74534-104"}]: dispatch 2024-03-26T04:30:12.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:11 smithi016 ceph-mon[29101]: from='client.56693 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi016-74534-104"}]: dispatch 2024-03-26T04:30:12.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:11 smithi016 ceph-mon[29101]: pgmap v869: 228 pgs: 228 active+clean; 582 KiB data, 593 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:30:12.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:11 smithi016 ceph-mon[37215]: osdmap e566: 8 total, 8 up, 8 in 2024-03-26T04:30:12.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:11 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/396409920' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi016-74534-104"}]: dispatch 2024-03-26T04:30:12.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:11 smithi016 ceph-mon[37215]: from='client.56693 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi016-74534-104"}]: dispatch 2024-03-26T04:30:12.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:11 smithi016 ceph-mon[37215]: pgmap v869: 228 pgs: 228 active+clean; 582 KiB data, 593 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:30:13.041 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:30:12 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:30:13.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:12 smithi183 ceph-mon[33745]: from='client.56693 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi016-74534-104"}]': finished 2024-03-26T04:30:13.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:12 smithi183 ceph-mon[33745]: osdmap e567: 8 total, 8 up, 8 in 2024-03-26T04:30:13.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:12 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/396409920' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi016-74534-104"}]: dispatch 2024-03-26T04:30:13.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:12 smithi183 ceph-mon[33745]: from='client.56693 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi016-74534-104"}]: dispatch 2024-03-26T04:30:13.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:12 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:30:13.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:12 smithi016 ceph-mon[37215]: from='client.56693 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi016-74534-104"}]': finished 2024-03-26T04:30:13.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:12 smithi016 ceph-mon[37215]: osdmap e567: 8 total, 8 up, 8 in 2024-03-26T04:30:13.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:12 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/396409920' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi016-74534-104"}]: dispatch 2024-03-26T04:30:13.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:12 smithi016 ceph-mon[37215]: from='client.56693 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi016-74534-104"}]: dispatch 2024-03-26T04:30:13.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:12 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:30:13.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:12 smithi016 ceph-mon[29101]: from='client.56693 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosTierECPP_smithi016-74534-104"}]': finished 2024-03-26T04:30:13.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:12 smithi016 ceph-mon[29101]: osdmap e567: 8 total, 8 up, 8 in 2024-03-26T04:30:13.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:12 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/396409920' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi016-74534-104"}]: dispatch 2024-03-26T04:30:13.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:12 smithi016 ceph-mon[29101]: from='client.56693 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi016-74534-104"}]: dispatch 2024-03-26T04:30:13.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:12 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:30:14.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:13 smithi183 ceph-mon[33745]: from='client.56693 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi016-74534-104"}]': finished 2024-03-26T04:30:14.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:13 smithi183 ceph-mon[33745]: osdmap e568: 8 total, 8 up, 8 in 2024-03-26T04:30:14.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:13 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:30:14.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:13 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:30:14.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:13 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:30:14.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:13 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:30:14.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:13 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi016-74534-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:30:14.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:13 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi016-74534-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:30:14.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:13 smithi183 ceph-mon[33745]: pgmap v872: 228 pgs: 228 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T04:30:14.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:13 smithi016 ceph-mon[37215]: from='client.56693 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi016-74534-104"}]': finished 2024-03-26T04:30:14.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:13 smithi016 ceph-mon[37215]: osdmap e568: 8 total, 8 up, 8 in 2024-03-26T04:30:14.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:13 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:30:14.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:13 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:30:14.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:13 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:30:14.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:13 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:30:14.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:13 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi016-74534-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:30:14.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:13 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi016-74534-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:30:14.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:13 smithi016 ceph-mon[37215]: pgmap v872: 228 pgs: 228 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T04:30:14.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:13 smithi016 ceph-mon[29101]: from='client.56693 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosTierECPP_smithi016-74534-104"}]': finished 2024-03-26T04:30:14.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:13 smithi016 ceph-mon[29101]: osdmap e568: 8 total, 8 up, 8 in 2024-03-26T04:30:14.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:13 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:30:14.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:13 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:30:14.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:13 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:30:14.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:13 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:30:14.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:13 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi016-74534-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:30:14.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:13 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi016-74534-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-03-26T04:30:14.164 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:13 smithi016 ceph-mon[29101]: pgmap v872: 228 pgs: 228 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 op/s 2024-03-26T04:30:15.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:14 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi016-74534-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:30:15.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:14 smithi183 ceph-mon[33745]: osdmap e569: 8 total, 8 up, 8 in 2024-03-26T04:30:15.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:14 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi016-74534-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:30:15.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:14 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi016-74534-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:30:15.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:14 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi016-74534-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:30:15.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:14 smithi016 ceph-mon[37215]: osdmap e569: 8 total, 8 up, 8 in 2024-03-26T04:30:15.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:14 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi016-74534-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:30:15.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:14 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi016-74534-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:30:15.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:14 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-test-rados-api-smithi016-74534-111", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-03-26T04:30:15.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:14 smithi016 ceph-mon[29101]: osdmap e569: 8 total, 8 up, 8 in 2024-03-26T04:30:15.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:14 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi016-74534-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:30:15.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:14 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "test-rados-api-smithi016-74534-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:30:16.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:15 smithi183 ceph-mon[33745]: osdmap e570: 8 total, 8 up, 8 in 2024-03-26T04:30:16.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:15 smithi183 ceph-mon[33745]: pgmap v875: 228 pgs: 228 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:30:16.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:15 smithi016 ceph-mon[29101]: osdmap e570: 8 total, 8 up, 8 in 2024-03-26T04:30:16.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:15 smithi016 ceph-mon[29101]: pgmap v875: 228 pgs: 228 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:30:16.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:15 smithi016 ceph-mon[37215]: osdmap e570: 8 total, 8 up, 8 in 2024-03-26T04:30:16.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:15 smithi016 ceph-mon[37215]: pgmap v875: 228 pgs: 228 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:30:17.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:16 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi016-74534-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:30:17.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:16 smithi183 ceph-mon[33745]: osdmap e571: 8 total, 8 up, 8 in 2024-03-26T04:30:17.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:16 smithi183 ceph-mon[33745]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:30:17.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:16 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi016-74534-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:30:17.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:16 smithi016 ceph-mon[37215]: osdmap e571: 8 total, 8 up, 8 in 2024-03-26T04:30:17.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:16 smithi016 ceph-mon[37215]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:30:17.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:16 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "test-rados-api-smithi016-74534-111", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:30:17.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:16 smithi016 ceph-mon[29101]: osdmap e571: 8 total, 8 up, 8 in 2024-03-26T04:30:17.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:16 smithi016 ceph-mon[29101]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:30:18.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:17 smithi183 ceph-mon[33745]: osdmap e572: 8 total, 8 up, 8 in 2024-03-26T04:30:18.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:17 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:30:18.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:17 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:30:18.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:17 smithi183 ceph-mon[33745]: pgmap v878: 268 pgs: 23 creating+peering, 13 unknown, 232 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:30:18.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:17 smithi016 ceph-mon[29101]: osdmap e572: 8 total, 8 up, 8 in 2024-03-26T04:30:18.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:17 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:30:18.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:17 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:30:18.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:17 smithi016 ceph-mon[29101]: pgmap v878: 268 pgs: 23 creating+peering, 13 unknown, 232 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:30:18.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:17 smithi016 ceph-mon[37215]: osdmap e572: 8 total, 8 up, 8 in 2024-03-26T04:30:18.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:17 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:30:18.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:17 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-112","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:30:18.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:17 smithi016 ceph-mon[37215]: pgmap v878: 268 pgs: 23 creating+peering, 13 unknown, 232 active+clean; 582 KiB data, 598 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:30:19.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:18 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:30:19.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:18 smithi183 ceph-mon[33745]: osdmap e573: 8 total, 8 up, 8 in 2024-03-26T04:30:19.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:18 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:30:19.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:18 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:30:19.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:18 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:30:19.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:18 smithi016 ceph-mon[29101]: osdmap e573: 8 total, 8 up, 8 in 2024-03-26T04:30:19.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:18 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:30:19.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:18 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:30:19.163 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:30:18 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:30:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:30:19.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:18 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-112","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:30:19.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:18 smithi016 ceph-mon[37215]: osdmap e573: 8 total, 8 up, 8 in 2024-03-26T04:30:19.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:18 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:30:19.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:18 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-112", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:30:20.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:19 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-112", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:30:20.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:19 smithi183 ceph-mon[33745]: osdmap e574: 8 total, 8 up, 8 in 2024-03-26T04:30:20.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:19 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-112"}]: dispatch 2024-03-26T04:30:20.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:19 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-112"}]: dispatch 2024-03-26T04:30:20.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:19 smithi183 ceph-mon[33745]: pgmap v881: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:30:20.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:19 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-112", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:30:20.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:19 smithi016 ceph-mon[29101]: osdmap e574: 8 total, 8 up, 8 in 2024-03-26T04:30:20.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:19 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-112"}]: dispatch 2024-03-26T04:30:20.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:19 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-112"}]: dispatch 2024-03-26T04:30:20.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:19 smithi016 ceph-mon[29101]: pgmap v881: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:30:20.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:19 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-112", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:30:20.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:19 smithi016 ceph-mon[37215]: osdmap e574: 8 total, 8 up, 8 in 2024-03-26T04:30:20.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:19 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-112"}]: dispatch 2024-03-26T04:30:20.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:19 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-112"}]: dispatch 2024-03-26T04:30:20.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:19 smithi016 ceph-mon[37215]: pgmap v881: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 599 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:30:21.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:20 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-112"}]': finished 2024-03-26T04:30:21.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:20 smithi183 ceph-mon[33745]: osdmap e575: 8 total, 8 up, 8 in 2024-03-26T04:30:21.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:20 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:30:21.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:20 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:30:21.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:20 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-112"}]': finished 2024-03-26T04:30:21.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:20 smithi016 ceph-mon[29101]: osdmap e575: 8 total, 8 up, 8 in 2024-03-26T04:30:21.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:20 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:30:21.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:20 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:30:21.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:20 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-112"}]': finished 2024-03-26T04:30:21.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:20 smithi016 ceph-mon[37215]: osdmap e575: 8 total, 8 up, 8 in 2024-03-26T04:30:21.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:20 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:30:21.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:20 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:30:22.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:21 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:30:22.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:21 smithi183 ceph-mon[33745]: osdmap e576: 8 total, 8 up, 8 in 2024-03-26T04:30:22.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:21 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-112"}]: dispatch 2024-03-26T04:30:22.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:21 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-112"}]: dispatch 2024-03-26T04:30:22.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:21 smithi183 ceph-mon[33745]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:30:22.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:21 smithi183 ceph-mon[33745]: pgmap v884: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 599 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:30:22.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:21 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-112"}]': finished 2024-03-26T04:30:22.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:21 smithi183 ceph-mon[33745]: osdmap e577: 8 total, 8 up, 8 in 2024-03-26T04:30:22.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:21 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:30:22.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:21 smithi016 ceph-mon[29101]: osdmap e576: 8 total, 8 up, 8 in 2024-03-26T04:30:22.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:21 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-112"}]: dispatch 2024-03-26T04:30:22.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:21 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-112"}]: dispatch 2024-03-26T04:30:22.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:21 smithi016 ceph-mon[29101]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:30:22.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:21 smithi016 ceph-mon[29101]: pgmap v884: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 599 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:30:22.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:21 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-112"}]': finished 2024-03-26T04:30:22.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:21 smithi016 ceph-mon[29101]: osdmap e577: 8 total, 8 up, 8 in 2024-03-26T04:30:22.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:21 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:30:22.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:21 smithi016 ceph-mon[37215]: osdmap e576: 8 total, 8 up, 8 in 2024-03-26T04:30:22.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:21 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-112"}]: dispatch 2024-03-26T04:30:22.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:21 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-112"}]: dispatch 2024-03-26T04:30:22.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:21 smithi016 ceph-mon[37215]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:30:22.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:21 smithi016 ceph-mon[37215]: pgmap v884: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 599 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:30:22.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:21 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-112"}]': finished 2024-03-26T04:30:22.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:21 smithi016 ceph-mon[37215]: osdmap e577: 8 total, 8 up, 8 in 2024-03-26T04:30:23.042 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:30:22 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:30:23.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:22 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:30:23.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:22 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:30:23.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:22 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:30:23.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:22 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:30:23.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:22 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:30:23.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:22 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:30:24.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:23 smithi183 ceph-mon[33745]: osdmap e578: 8 total, 8 up, 8 in 2024-03-26T04:30:24.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:23 smithi183 ceph-mon[33745]: pgmap v887: 236 pgs: 236 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:30:24.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:23 smithi016 ceph-mon[37215]: osdmap e578: 8 total, 8 up, 8 in 2024-03-26T04:30:24.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:23 smithi016 ceph-mon[37215]: pgmap v887: 236 pgs: 236 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:30:24.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:23 smithi016 ceph-mon[29101]: osdmap e578: 8 total, 8 up, 8 in 2024-03-26T04:30:24.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:23 smithi016 ceph-mon[29101]: pgmap v887: 236 pgs: 236 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:30:25.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:24 smithi016 ceph-mon[29101]: osdmap e579: 8 total, 8 up, 8 in 2024-03-26T04:30:25.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:24 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:30:25.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:24 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:30:25.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:24 smithi016 ceph-mon[37215]: osdmap e579: 8 total, 8 up, 8 in 2024-03-26T04:30:25.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:24 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:30:25.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:24 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:30:25.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:24 smithi183 ceph-mon[33745]: osdmap e579: 8 total, 8 up, 8 in 2024-03-26T04:30:25.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:24 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:30:25.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:24 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-114","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:30:26.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:25 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:30:26.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:25 smithi016 ceph-mon[29101]: osdmap e580: 8 total, 8 up, 8 in 2024-03-26T04:30:26.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:25 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:30:26.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:25 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:30:26.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:25 smithi016 ceph-mon[29101]: pgmap v890: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:30:26.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:25 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-114", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:30:26.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:25 smithi016 ceph-mon[29101]: osdmap e581: 8 total, 8 up, 8 in 2024-03-26T04:30:26.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:25 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-114"}]: dispatch 2024-03-26T04:30:26.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:25 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-114"}]: dispatch 2024-03-26T04:30:26.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:25 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:30:26.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:25 smithi016 ceph-mon[37215]: osdmap e580: 8 total, 8 up, 8 in 2024-03-26T04:30:26.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:25 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:30:26.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:25 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:30:26.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:25 smithi016 ceph-mon[37215]: pgmap v890: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:30:26.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:25 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-114", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:30:26.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:25 smithi016 ceph-mon[37215]: osdmap e581: 8 total, 8 up, 8 in 2024-03-26T04:30:26.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:25 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-114"}]: dispatch 2024-03-26T04:30:26.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:25 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-114"}]: dispatch 2024-03-26T04:30:26.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:25 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-114","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:30:26.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:25 smithi183 ceph-mon[33745]: osdmap e580: 8 total, 8 up, 8 in 2024-03-26T04:30:26.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:25 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:30:26.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:25 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-114", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:30:26.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:25 smithi183 ceph-mon[33745]: pgmap v890: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 603 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:30:26.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:25 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-114", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:30:26.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:25 smithi183 ceph-mon[33745]: osdmap e581: 8 total, 8 up, 8 in 2024-03-26T04:30:26.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:25 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-114"}]: dispatch 2024-03-26T04:30:26.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:25 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-114"}]: dispatch 2024-03-26T04:30:28.094 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:27 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-114"}]': finished 2024-03-26T04:30:28.094 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:27 smithi183 ceph-mon[33745]: osdmap e582: 8 total, 8 up, 8 in 2024-03-26T04:30:28.094 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:27 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-114", "mode": "writeback"}]: dispatch 2024-03-26T04:30:28.094 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:27 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-114", "mode": "writeback"}]: dispatch 2024-03-26T04:30:28.094 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:27 smithi183 ceph-mon[33745]: pgmap v893: 268 pgs: 13 unknown, 255 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:30:28.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:27 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-114"}]': finished 2024-03-26T04:30:28.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:27 smithi016 ceph-mon[29101]: osdmap e582: 8 total, 8 up, 8 in 2024-03-26T04:30:28.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:27 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-114", "mode": "writeback"}]: dispatch 2024-03-26T04:30:28.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:27 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-114", "mode": "writeback"}]: dispatch 2024-03-26T04:30:28.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:27 smithi016 ceph-mon[29101]: pgmap v893: 268 pgs: 13 unknown, 255 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:30:28.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:27 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-114"}]': finished 2024-03-26T04:30:28.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:27 smithi016 ceph-mon[37215]: osdmap e582: 8 total, 8 up, 8 in 2024-03-26T04:30:28.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:27 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-114", "mode": "writeback"}]: dispatch 2024-03-26T04:30:28.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:27 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-114", "mode": "writeback"}]: dispatch 2024-03-26T04:30:28.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:27 smithi016 ceph-mon[37215]: pgmap v893: 268 pgs: 13 unknown, 255 active+clean; 582 KiB data, 604 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:30:29.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:30:28 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:30:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:30:29.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:28 smithi016 ceph-mon[29101]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:30:29.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:28 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-114", "mode": "writeback"}]': finished 2024-03-26T04:30:29.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:28 smithi016 ceph-mon[29101]: osdmap e583: 8 total, 8 up, 8 in 2024-03-26T04:30:29.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:28 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:30:29.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:28 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:30:29.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:28 smithi016 ceph-mon[37215]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:30:29.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:28 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-114", "mode": "writeback"}]': finished 2024-03-26T04:30:29.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:28 smithi016 ceph-mon[37215]: osdmap e583: 8 total, 8 up, 8 in 2024-03-26T04:30:29.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:28 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:30:29.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:28 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:30:29.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:28 smithi183 ceph-mon[33745]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:30:29.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:28 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-114", "mode": "writeback"}]': finished 2024-03-26T04:30:29.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:28 smithi183 ceph-mon[33745]: osdmap e583: 8 total, 8 up, 8 in 2024-03-26T04:30:29.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:28 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:30:29.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:28 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:30:30.165 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:29 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:30:30.165 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:29 smithi016 ceph-mon[29101]: osdmap e584: 8 total, 8 up, 8 in 2024-03-26T04:30:30.166 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:29 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-114"}]: dispatch 2024-03-26T04:30:30.166 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:29 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-114"}]: dispatch 2024-03-26T04:30:30.166 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:29 smithi016 ceph-mon[29101]: pgmap v896: 268 pgs: 268 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:30:30.166 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:29 smithi016 ceph-mon[29101]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:30:30.166 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:29 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-114"}]': finished 2024-03-26T04:30:30.166 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:29 smithi016 ceph-mon[29101]: osdmap e585: 8 total, 8 up, 8 in 2024-03-26T04:30:30.166 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:29 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:30:30.166 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:29 smithi016 ceph-mon[37215]: osdmap e584: 8 total, 8 up, 8 in 2024-03-26T04:30:30.166 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:29 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-114"}]: dispatch 2024-03-26T04:30:30.167 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:29 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-114"}]: dispatch 2024-03-26T04:30:30.167 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:29 smithi016 ceph-mon[37215]: pgmap v896: 268 pgs: 268 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:30:30.167 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:29 smithi016 ceph-mon[37215]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:30:30.167 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:29 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-114"}]': finished 2024-03-26T04:30:30.167 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:29 smithi016 ceph-mon[37215]: osdmap e585: 8 total, 8 up, 8 in 2024-03-26T04:30:30.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:29 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:30:30.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:29 smithi183 ceph-mon[33745]: osdmap e584: 8 total, 8 up, 8 in 2024-03-26T04:30:30.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:29 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-114"}]: dispatch 2024-03-26T04:30:30.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:29 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-114"}]: dispatch 2024-03-26T04:30:30.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:29 smithi183 ceph-mon[33745]: pgmap v896: 268 pgs: 268 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:30:30.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:29 smithi183 ceph-mon[33745]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:30:30.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:29 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-114"}]': finished 2024-03-26T04:30:30.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:29 smithi183 ceph-mon[33745]: osdmap e585: 8 total, 8 up, 8 in 2024-03-26T04:30:32.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:31 smithi016 ceph-mon[29101]: osdmap e586: 8 total, 8 up, 8 in 2024-03-26T04:30:32.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:31 smithi016 ceph-mon[29101]: pgmap v899: 236 pgs: 236 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-03-26T04:30:32.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:31 smithi016 ceph-mon[37215]: osdmap e586: 8 total, 8 up, 8 in 2024-03-26T04:30:32.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:31 smithi016 ceph-mon[37215]: pgmap v899: 236 pgs: 236 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-03-26T04:30:32.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:31 smithi183 ceph-mon[33745]: osdmap e586: 8 total, 8 up, 8 in 2024-03-26T04:30:32.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:31 smithi183 ceph-mon[33745]: pgmap v899: 236 pgs: 236 active+clean; 582 KiB data, 608 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-03-26T04:30:33.041 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:30:32 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:30:33.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:32 smithi183 ceph-mon[33745]: osdmap e587: 8 total, 8 up, 8 in 2024-03-26T04:30:33.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:32 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:30:33.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:32 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:30:33.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:32 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:30:33.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:32 smithi016 ceph-mon[37215]: osdmap e587: 8 total, 8 up, 8 in 2024-03-26T04:30:33.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:32 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:30:33.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:32 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:30:33.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:32 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:30:33.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:32 smithi016 ceph-mon[29101]: osdmap e587: 8 total, 8 up, 8 in 2024-03-26T04:30:33.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:32 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:30:33.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:32 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-116","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:30:33.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:32 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:30:34.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:33 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:30:34.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:33 smithi016 ceph-mon[29101]: osdmap e588: 8 total, 8 up, 8 in 2024-03-26T04:30:34.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:33 smithi016 ceph-mon[29101]: pgmap v902: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:30:34.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:33 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:30:34.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:33 smithi016 ceph-mon[37215]: osdmap e588: 8 total, 8 up, 8 in 2024-03-26T04:30:34.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:33 smithi016 ceph-mon[37215]: pgmap v902: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:30:34.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:33 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-116","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:30:34.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:33 smithi183 ceph-mon[33745]: osdmap e588: 8 total, 8 up, 8 in 2024-03-26T04:30:34.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:33 smithi183 ceph-mon[33745]: pgmap v902: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:30:35.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:34 smithi016 ceph-mon[29101]: osdmap e589: 8 total, 8 up, 8 in 2024-03-26T04:30:35.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:34 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:30:35.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:34 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:30:35.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:34 smithi016 ceph-mon[37215]: osdmap e589: 8 total, 8 up, 8 in 2024-03-26T04:30:35.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:34 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:30:35.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:34 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:30:35.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:34 smithi183 ceph-mon[33745]: osdmap e589: 8 total, 8 up, 8 in 2024-03-26T04:30:35.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:34 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:30:35.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:34 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-116", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:30:36.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:35 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-116", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:30:36.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:35 smithi016 ceph-mon[37215]: osdmap e590: 8 total, 8 up, 8 in 2024-03-26T04:30:36.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:35 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-116"}]: dispatch 2024-03-26T04:30:36.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:35 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-116"}]: dispatch 2024-03-26T04:30:36.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:35 smithi016 ceph-mon[37215]: pgmap v905: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:30:36.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:35 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-116"}]': finished 2024-03-26T04:30:36.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:35 smithi016 ceph-mon[37215]: osdmap e591: 8 total, 8 up, 8 in 2024-03-26T04:30:36.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:35 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-116", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:30:36.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:35 smithi016 ceph-mon[29101]: osdmap e590: 8 total, 8 up, 8 in 2024-03-26T04:30:36.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:35 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-116"}]: dispatch 2024-03-26T04:30:36.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:35 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-116"}]: dispatch 2024-03-26T04:30:36.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:35 smithi016 ceph-mon[29101]: pgmap v905: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:30:36.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:35 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-116"}]': finished 2024-03-26T04:30:36.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:35 smithi016 ceph-mon[29101]: osdmap e591: 8 total, 8 up, 8 in 2024-03-26T04:30:36.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:35 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-116", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:30:36.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:35 smithi183 ceph-mon[33745]: osdmap e590: 8 total, 8 up, 8 in 2024-03-26T04:30:36.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:35 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-116"}]: dispatch 2024-03-26T04:30:36.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:35 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-116"}]: dispatch 2024-03-26T04:30:36.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:35 smithi183 ceph-mon[33745]: pgmap v905: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:30:36.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:35 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-116"}]': finished 2024-03-26T04:30:36.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:35 smithi183 ceph-mon[33745]: osdmap e591: 8 total, 8 up, 8 in 2024-03-26T04:30:37.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:36 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-116", "mode": "writeback"}]: dispatch 2024-03-26T04:30:37.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:36 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-116", "mode": "writeback"}]: dispatch 2024-03-26T04:30:37.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:36 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:30:37.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:36 smithi016 ceph-mon[37215]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:30:37.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:36 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-116", "mode": "writeback"}]': finished 2024-03-26T04:30:37.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:36 smithi016 ceph-mon[37215]: osdmap e592: 8 total, 8 up, 8 in 2024-03-26T04:30:37.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:36 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-116", "mode": "writeback"}]: dispatch 2024-03-26T04:30:37.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:36 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-116", "mode": "writeback"}]: dispatch 2024-03-26T04:30:37.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:36 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:30:37.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:36 smithi016 ceph-mon[29101]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:30:37.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:36 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-116", "mode": "writeback"}]': finished 2024-03-26T04:30:37.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:36 smithi016 ceph-mon[29101]: osdmap e592: 8 total, 8 up, 8 in 2024-03-26T04:30:37.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:36 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-116", "mode": "writeback"}]: dispatch 2024-03-26T04:30:37.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:36 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-116", "mode": "writeback"}]: dispatch 2024-03-26T04:30:37.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:36 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:30:37.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:36 smithi183 ceph-mon[33745]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:30:37.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:36 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-116", "mode": "writeback"}]': finished 2024-03-26T04:30:37.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:36 smithi183 ceph-mon[33745]: osdmap e592: 8 total, 8 up, 8 in 2024-03-26T04:30:38.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:37 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "311.7"}]: dispatch 2024-03-26T04:30:38.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:37 smithi016 ceph-mon[29101]: from='mon.? v1:172.21.15.16:0/2907881663' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "311.7"}]: dispatch 2024-03-26T04:30:38.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:37 smithi016 ceph-mon[29101]: pgmap v908: 268 pgs: 17 creating+peering, 251 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T04:30:38.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:37 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:30:38.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:37 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:30:38.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:37 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "311.7"}]: dispatch 2024-03-26T04:30:38.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:37 smithi016 ceph-mon[37215]: from='mon.? v1:172.21.15.16:0/2907881663' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "311.7"}]: dispatch 2024-03-26T04:30:38.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:37 smithi016 ceph-mon[37215]: pgmap v908: 268 pgs: 17 creating+peering, 251 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T04:30:38.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:37 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:30:38.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:37 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:30:38.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:37 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "pg scrub", "pgid": "311.7"}]: dispatch 2024-03-26T04:30:38.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:37 smithi183 ceph-mon[33745]: from='mon.? v1:172.21.15.16:0/2907881663' entity='mon.' cmd=[{"prefix": "pg scrub", "pgid": "311.7"}]: dispatch 2024-03-26T04:30:38.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:37 smithi183 ceph-mon[33745]: pgmap v908: 268 pgs: 17 creating+peering, 251 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T04:30:38.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:37 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:30:38.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:37 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:30:39.140 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:38 smithi183 ceph-mon[33745]: 311.7 deep-scrub starts 2024-03-26T04:30:39.140 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:38 smithi183 ceph-mon[33745]: 311.7 deep-scrub ok 2024-03-26T04:30:39.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:30:38 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:30:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:30:39.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:38 smithi016 ceph-mon[37215]: 311.7 deep-scrub starts 2024-03-26T04:30:39.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:38 smithi016 ceph-mon[37215]: 311.7 deep-scrub ok 2024-03-26T04:30:39.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:38 smithi016 ceph-mon[29101]: 311.7 deep-scrub starts 2024-03-26T04:30:39.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:38 smithi016 ceph-mon[29101]: 311.7 deep-scrub ok 2024-03-26T04:30:40.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:39 smithi183 ceph-mon[33745]: pgmap v909: 268 pgs: 268 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-26T04:30:40.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:39 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:30:40.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:39 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:30:40.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:39 smithi016 ceph-mon[29101]: pgmap v909: 268 pgs: 268 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-26T04:30:40.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:39 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:30:40.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:39 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:30:40.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:39 smithi016 ceph-mon[37215]: pgmap v909: 268 pgs: 268 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.3 KiB/s wr, 3 op/s 2024-03-26T04:30:40.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:39 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:30:40.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:39 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:30:42.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:41 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:30:42.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:41 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:30:42.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:41 smithi016 ceph-mon[29101]: pgmap v910: 268 pgs: 268 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-03-26T04:30:42.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:41 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:30:42.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:41 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:30:42.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:41 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:30:42.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:41 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:30:42.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:41 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:30:42.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:41 smithi016 ceph-mon[37215]: pgmap v910: 268 pgs: 268 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-03-26T04:30:42.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:41 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:30:42.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:41 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:30:42.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:41 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:30:42.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:41 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:30:42.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:41 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:30:42.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:41 smithi183 ceph-mon[33745]: pgmap v910: 268 pgs: 268 active+clean; 582 KiB data, 609 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-03-26T04:30:42.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:41 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:30:42.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:41 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:30:42.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:41 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:30:43.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:42 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:30:43.042 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:30:42 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:30:43.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:42 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:30:43.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:42 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:30:44.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:43 smithi016 ceph-mon[37215]: pgmap v911: 268 pgs: 268 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 895 B/s wr, 2 op/s 2024-03-26T04:30:44.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:43 smithi016 ceph-mon[29101]: pgmap v911: 268 pgs: 268 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 895 B/s wr, 2 op/s 2024-03-26T04:30:44.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:43 smithi183 ceph-mon[33745]: pgmap v911: 268 pgs: 268 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 895 B/s wr, 2 op/s 2024-03-26T04:30:46.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:46 smithi183 ceph-mon[33745]: pgmap v912: 268 pgs: 268 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 765 B/s wr, 2 op/s 2024-03-26T04:30:46.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:46 smithi016 ceph-mon[37215]: pgmap v912: 268 pgs: 268 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 765 B/s wr, 2 op/s 2024-03-26T04:30:46.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:46 smithi016 ceph-mon[29101]: pgmap v912: 268 pgs: 268 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 765 B/s wr, 2 op/s 2024-03-26T04:30:48.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:48 smithi183 ceph-mon[33745]: pgmap v913: 268 pgs: 268 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 691 B/s wr, 2 op/s 2024-03-26T04:30:48.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:48 smithi016 ceph-mon[29101]: pgmap v913: 268 pgs: 268 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 691 B/s wr, 2 op/s 2024-03-26T04:30:48.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:48 smithi016 ceph-mon[37215]: pgmap v913: 268 pgs: 268 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 691 B/s wr, 2 op/s 2024-03-26T04:30:49.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:30:48 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:30:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:30:50.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:50 smithi183 ceph-mon[33745]: pgmap v914: 268 pgs: 268 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 597 B/s wr, 1 op/s 2024-03-26T04:30:50.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:50 smithi016 ceph-mon[29101]: pgmap v914: 268 pgs: 268 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 597 B/s wr, 1 op/s 2024-03-26T04:30:50.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:50 smithi016 ceph-mon[37215]: pgmap v914: 268 pgs: 268 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 938 B/s rd, 597 B/s wr, 1 op/s 2024-03-26T04:30:52.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:52 smithi183 ceph-mon[33745]: pgmap v915: 268 pgs: 268 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:30:52.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:52 smithi016 ceph-mon[37215]: pgmap v915: 268 pgs: 268 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:30:52.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:52 smithi016 ceph-mon[29101]: pgmap v915: 268 pgs: 268 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:30:53.041 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:30:52 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:30:53.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:53 smithi183 ceph-mon[33745]: osdmap e593: 8 total, 8 up, 8 in 2024-03-26T04:30:53.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:53 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:30:53.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:53 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:30:53.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:53 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:30:53.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:53 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:30:53.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:53 smithi016 ceph-mon[37215]: osdmap e593: 8 total, 8 up, 8 in 2024-03-26T04:30:53.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:53 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:30:53.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:53 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:30:53.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:53 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:30:53.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:53 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:30:53.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:53 smithi016 ceph-mon[29101]: osdmap e593: 8 total, 8 up, 8 in 2024-03-26T04:30:53.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:53 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:30:53.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:53 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:30:53.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:53 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:30:53.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:53 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:30:54.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:54 smithi183 ceph-mon[33745]: pgmap v917: 268 pgs: 268 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T04:30:54.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:54 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:30:54.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:54 smithi183 ceph-mon[33745]: osdmap e594: 8 total, 8 up, 8 in 2024-03-26T04:30:54.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:54 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-116"}]: dispatch 2024-03-26T04:30:54.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:54 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-116"}]: dispatch 2024-03-26T04:30:54.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:54 smithi183 ceph-mon[33745]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:30:54.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:54 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-116"}]': finished 2024-03-26T04:30:54.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:54 smithi183 ceph-mon[33745]: osdmap e595: 8 total, 8 up, 8 in 2024-03-26T04:30:54.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:54 smithi016 ceph-mon[29101]: pgmap v917: 268 pgs: 268 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T04:30:54.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:54 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:30:54.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:54 smithi016 ceph-mon[29101]: osdmap e594: 8 total, 8 up, 8 in 2024-03-26T04:30:54.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:54 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-116"}]: dispatch 2024-03-26T04:30:54.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:54 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-116"}]: dispatch 2024-03-26T04:30:54.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:54 smithi016 ceph-mon[29101]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:30:54.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:54 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-116"}]': finished 2024-03-26T04:30:54.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:54 smithi016 ceph-mon[29101]: osdmap e595: 8 total, 8 up, 8 in 2024-03-26T04:30:54.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:54 smithi016 ceph-mon[37215]: pgmap v917: 268 pgs: 268 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-26T04:30:54.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:54 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:30:54.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:54 smithi016 ceph-mon[37215]: osdmap e594: 8 total, 8 up, 8 in 2024-03-26T04:30:54.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:54 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-116"}]: dispatch 2024-03-26T04:30:54.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:54 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-116"}]: dispatch 2024-03-26T04:30:54.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:54 smithi016 ceph-mon[37215]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:30:54.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:54 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-116"}]': finished 2024-03-26T04:30:54.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:54 smithi016 ceph-mon[37215]: osdmap e595: 8 total, 8 up, 8 in 2024-03-26T04:30:55.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:55 smithi016 ceph-mon[29101]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:30:55.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:55 smithi016 ceph-mon[29101]: osdmap e596: 8 total, 8 up, 8 in 2024-03-26T04:30:55.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:55 smithi016 ceph-mon[37215]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:30:55.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:55 smithi016 ceph-mon[37215]: osdmap e596: 8 total, 8 up, 8 in 2024-03-26T04:30:56.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:55 smithi183 ceph-mon[33745]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:30:56.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:55 smithi183 ceph-mon[33745]: osdmap e596: 8 total, 8 up, 8 in 2024-03-26T04:30:56.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:56 smithi016 ceph-mon[29101]: pgmap v920: 268 pgs: 268 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T04:30:56.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:56 smithi016 ceph-mon[29101]: osdmap e597: 8 total, 8 up, 8 in 2024-03-26T04:30:56.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:56 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:30:56.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:56 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:30:56.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:56 smithi016 ceph-mon[37215]: pgmap v920: 268 pgs: 268 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T04:30:56.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:56 smithi016 ceph-mon[37215]: osdmap e597: 8 total, 8 up, 8 in 2024-03-26T04:30:56.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:56 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:30:56.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:56 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:30:57.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:56 smithi183 ceph-mon[33745]: pgmap v920: 268 pgs: 268 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T04:30:57.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:56 smithi183 ceph-mon[33745]: osdmap e597: 8 total, 8 up, 8 in 2024-03-26T04:30:57.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:56 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:30:57.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:56 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-118","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:30:58.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:58 smithi016 ceph-mon[37215]: pgmap v923: 268 pgs: 23 creating+peering, 9 unknown, 236 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:30:58.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:58 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:30:58.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:58 smithi016 ceph-mon[37215]: osdmap e598: 8 total, 8 up, 8 in 2024-03-26T04:30:58.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:58 smithi016 ceph-mon[29101]: pgmap v923: 268 pgs: 23 creating+peering, 9 unknown, 236 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:30:58.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:58 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:30:58.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:58 smithi016 ceph-mon[29101]: osdmap e598: 8 total, 8 up, 8 in 2024-03-26T04:30:58.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:58 smithi183 ceph-mon[33745]: pgmap v923: 268 pgs: 23 creating+peering, 9 unknown, 236 active+clean; 582 KiB data, 610 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:30:58.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:58 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-118","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:30:58.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:58 smithi183 ceph-mon[33745]: osdmap e598: 8 total, 8 up, 8 in 2024-03-26T04:30:59.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:30:58 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:30:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:30:59.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:59 smithi016 ceph-mon[37215]: osdmap e599: 8 total, 8 up, 8 in 2024-03-26T04:30:59.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:59 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:30:59.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:30:59 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:30:59.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:59 smithi016 ceph-mon[29101]: osdmap e599: 8 total, 8 up, 8 in 2024-03-26T04:30:59.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:59 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:30:59.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:30:59 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:30:59.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:59 smithi183 ceph-mon[33745]: osdmap e599: 8 total, 8 up, 8 in 2024-03-26T04:30:59.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:59 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:30:59.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:30:59 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-118", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:31:00.565 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:00 smithi016 ceph-mon[37215]: pgmap v926: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:31:00.565 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:00 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-118", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:31:00.565 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:00 smithi016 ceph-mon[37215]: osdmap e600: 8 total, 8 up, 8 in 2024-03-26T04:31:00.565 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:00 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-118"}]: dispatch 2024-03-26T04:31:00.565 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:00 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-118"}]: dispatch 2024-03-26T04:31:00.565 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:00 smithi016 ceph-mon[29101]: pgmap v926: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:31:00.565 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:00 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-118", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:31:00.565 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:00 smithi016 ceph-mon[29101]: osdmap e600: 8 total, 8 up, 8 in 2024-03-26T04:31:00.565 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:00 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-118"}]: dispatch 2024-03-26T04:31:00.565 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:00 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-118"}]: dispatch 2024-03-26T04:31:00.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:00 smithi183 ceph-mon[33745]: pgmap v926: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:31:00.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:00 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-118", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:31:00.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:00 smithi183 ceph-mon[33745]: osdmap e600: 8 total, 8 up, 8 in 2024-03-26T04:31:00.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:00 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-118"}]: dispatch 2024-03-26T04:31:00.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:00 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-118"}]: dispatch 2024-03-26T04:31:01.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:01 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-118"}]': finished 2024-03-26T04:31:01.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:01 smithi016 ceph-mon[29101]: osdmap e601: 8 total, 8 up, 8 in 2024-03-26T04:31:01.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:01 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-118", "mode": "writeback"}]: dispatch 2024-03-26T04:31:01.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:01 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-118", "mode": "writeback"}]: dispatch 2024-03-26T04:31:01.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:01 smithi016 ceph-mon[29101]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:31:01.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:01 smithi016 ceph-mon[29101]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:31:01.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:01 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-118", "mode": "writeback"}]': finished 2024-03-26T04:31:01.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:01 smithi016 ceph-mon[29101]: osdmap e602: 8 total, 8 up, 8 in 2024-03-26T04:31:01.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:01 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-118"}]': finished 2024-03-26T04:31:01.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:01 smithi016 ceph-mon[37215]: osdmap e601: 8 total, 8 up, 8 in 2024-03-26T04:31:01.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:01 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-118", "mode": "writeback"}]: dispatch 2024-03-26T04:31:01.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:01 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-118", "mode": "writeback"}]: dispatch 2024-03-26T04:31:01.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:01 smithi016 ceph-mon[37215]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:31:01.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:01 smithi016 ceph-mon[37215]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:31:01.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:01 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-118", "mode": "writeback"}]': finished 2024-03-26T04:31:01.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:01 smithi016 ceph-mon[37215]: osdmap e602: 8 total, 8 up, 8 in 2024-03-26T04:31:01.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:01 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-118"}]': finished 2024-03-26T04:31:01.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:01 smithi183 ceph-mon[33745]: osdmap e601: 8 total, 8 up, 8 in 2024-03-26T04:31:01.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:01 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-118", "mode": "writeback"}]: dispatch 2024-03-26T04:31:01.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:01 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-118", "mode": "writeback"}]: dispatch 2024-03-26T04:31:01.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:01 smithi183 ceph-mon[33745]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:31:01.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:01 smithi183 ceph-mon[33745]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:31:01.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:01 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-118", "mode": "writeback"}]': finished 2024-03-26T04:31:01.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:01 smithi183 ceph-mon[33745]: osdmap e602: 8 total, 8 up, 8 in 2024-03-26T04:31:02.625 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:02 smithi183 ceph-mon[33745]: pgmap v930: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:31:02.625 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:02 smithi183 ceph-mon[33745]: osdmap e603: 8 total, 8 up, 8 in 2024-03-26T04:31:02.625 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:02 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:31:02.625 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:02 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:31:02.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:02 smithi016 ceph-mon[37215]: pgmap v930: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:31:02.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:02 smithi016 ceph-mon[37215]: osdmap e603: 8 total, 8 up, 8 in 2024-03-26T04:31:02.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:02 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:31:02.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:02 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:31:02.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:02 smithi016 ceph-mon[29101]: pgmap v930: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 614 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:31:02.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:02 smithi016 ceph-mon[29101]: osdmap e603: 8 total, 8 up, 8 in 2024-03-26T04:31:02.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:02 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:31:02.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:02 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:31:03.041 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:31:02 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:31:03.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:03 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:31:03.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:03 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:31:03.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:03 smithi016 ceph-mon[37215]: osdmap e604: 8 total, 8 up, 8 in 2024-03-26T04:31:03.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:03 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-118"}]: dispatch 2024-03-26T04:31:03.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:03 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-118"}]: dispatch 2024-03-26T04:31:03.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:03 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:31:03.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:03 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:31:03.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:03 smithi016 ceph-mon[29101]: osdmap e604: 8 total, 8 up, 8 in 2024-03-26T04:31:03.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:03 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-118"}]: dispatch 2024-03-26T04:31:03.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:03 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-118"}]: dispatch 2024-03-26T04:31:03.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:03 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:31:03.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:03 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:31:03.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:03 smithi183 ceph-mon[33745]: osdmap e604: 8 total, 8 up, 8 in 2024-03-26T04:31:03.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:03 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-118"}]: dispatch 2024-03-26T04:31:03.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:03 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-118"}]: dispatch 2024-03-26T04:31:04.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:04 smithi016 ceph-mon[37215]: pgmap v933: 268 pgs: 268 active+clean; 582 KiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 524 B/s wr, 1 op/s 2024-03-26T04:31:04.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:04 smithi016 ceph-mon[37215]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:31:04.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:04 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-118"}]': finished 2024-03-26T04:31:04.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:04 smithi016 ceph-mon[37215]: osdmap e605: 8 total, 8 up, 8 in 2024-03-26T04:31:04.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:04 smithi016 ceph-mon[29101]: pgmap v933: 268 pgs: 268 active+clean; 582 KiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 524 B/s wr, 1 op/s 2024-03-26T04:31:04.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:04 smithi016 ceph-mon[29101]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:31:04.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:04 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-118"}]': finished 2024-03-26T04:31:04.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:04 smithi016 ceph-mon[29101]: osdmap e605: 8 total, 8 up, 8 in 2024-03-26T04:31:04.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:04 smithi183 ceph-mon[33745]: pgmap v933: 268 pgs: 268 active+clean; 582 KiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 524 B/s wr, 1 op/s 2024-03-26T04:31:04.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:04 smithi183 ceph-mon[33745]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:31:04.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:04 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-118"}]': finished 2024-03-26T04:31:04.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:04 smithi183 ceph-mon[33745]: osdmap e605: 8 total, 8 up, 8 in 2024-03-26T04:31:06.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:06 smithi183 ceph-mon[33745]: pgmap v935: 268 pgs: 268 active+clean; 582 KiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 509 B/s wr, 1 op/s 2024-03-26T04:31:06.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:06 smithi183 ceph-mon[33745]: osdmap e606: 8 total, 8 up, 8 in 2024-03-26T04:31:06.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:06 smithi016 ceph-mon[37215]: pgmap v935: 268 pgs: 268 active+clean; 582 KiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 509 B/s wr, 1 op/s 2024-03-26T04:31:06.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:06 smithi016 ceph-mon[37215]: osdmap e606: 8 total, 8 up, 8 in 2024-03-26T04:31:06.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:06 smithi016 ceph-mon[29101]: pgmap v935: 268 pgs: 268 active+clean; 582 KiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 509 B/s wr, 1 op/s 2024-03-26T04:31:06.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:06 smithi016 ceph-mon[29101]: osdmap e606: 8 total, 8 up, 8 in 2024-03-26T04:31:07.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:07 smithi183 ceph-mon[33745]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:31:07.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:07 smithi183 ceph-mon[33745]: osdmap e607: 8 total, 8 up, 8 in 2024-03-26T04:31:07.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:07 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:31:07.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:07 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:31:07.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:07 smithi016 ceph-mon[29101]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:31:07.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:07 smithi016 ceph-mon[29101]: osdmap e607: 8 total, 8 up, 8 in 2024-03-26T04:31:07.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:07 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:31:07.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:07 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:31:07.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:07 smithi016 ceph-mon[37215]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:31:07.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:07 smithi016 ceph-mon[37215]: osdmap e607: 8 total, 8 up, 8 in 2024-03-26T04:31:07.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:07 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:31:07.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:07 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-120","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:31:08.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:08 smithi183 ceph-mon[33745]: pgmap v938: 268 pgs: 24 creating+peering, 8 unknown, 236 active+clean; 582 KiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:31:08.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:08 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:31:08.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:08 smithi183 ceph-mon[33745]: osdmap e608: 8 total, 8 up, 8 in 2024-03-26T04:31:08.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:08 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:31:08.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:08 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:31:08.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:08 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:31:08.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:08 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:31:08.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:08 smithi016 ceph-mon[29101]: pgmap v938: 268 pgs: 24 creating+peering, 8 unknown, 236 active+clean; 582 KiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:31:08.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:08 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:31:08.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:08 smithi016 ceph-mon[29101]: osdmap e608: 8 total, 8 up, 8 in 2024-03-26T04:31:08.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:08 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:31:08.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:08 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:31:08.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:08 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:31:08.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:08 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:31:08.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:08 smithi016 ceph-mon[37215]: pgmap v938: 268 pgs: 24 creating+peering, 8 unknown, 236 active+clean; 582 KiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:31:08.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:08 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-120","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:31:08.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:08 smithi016 ceph-mon[37215]: osdmap e608: 8 total, 8 up, 8 in 2024-03-26T04:31:08.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:08 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:31:08.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:08 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-120", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:31:08.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:08 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:31:08.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:08 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:31:09.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:31:08 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:31:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:31:09.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:09 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-120", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:31:09.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:09 smithi183 ceph-mon[33745]: osdmap e609: 8 total, 8 up, 8 in 2024-03-26T04:31:09.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:09 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-120"}]: dispatch 2024-03-26T04:31:09.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:09 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-120"}]: dispatch 2024-03-26T04:31:09.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:09 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-120", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:31:09.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:09 smithi016 ceph-mon[37215]: osdmap e609: 8 total, 8 up, 8 in 2024-03-26T04:31:09.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:09 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-120"}]: dispatch 2024-03-26T04:31:09.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:09 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-120"}]: dispatch 2024-03-26T04:31:09.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:09 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-120", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:31:09.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:09 smithi016 ceph-mon[29101]: osdmap e609: 8 total, 8 up, 8 in 2024-03-26T04:31:09.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:09 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-120"}]: dispatch 2024-03-26T04:31:09.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:09 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-120"}]: dispatch 2024-03-26T04:31:10.497 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:10 smithi183 ceph-mon[33745]: pgmap v941: 268 pgs: 4 creating+activating, 24 creating+peering, 240 active+clean; 582 KiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:31:10.497 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:10 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-120"}]': finished 2024-03-26T04:31:10.498 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:10 smithi183 ceph-mon[33745]: osdmap e610: 8 total, 8 up, 8 in 2024-03-26T04:31:10.498 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:10 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-120", "mode": "writeback"}]: dispatch 2024-03-26T04:31:10.498 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:10 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-120", "mode": "writeback"}]: dispatch 2024-03-26T04:31:10.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:10 smithi016 ceph-mon[29101]: pgmap v941: 268 pgs: 4 creating+activating, 24 creating+peering, 240 active+clean; 582 KiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:31:10.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:10 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-120"}]': finished 2024-03-26T04:31:10.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:10 smithi016 ceph-mon[29101]: osdmap e610: 8 total, 8 up, 8 in 2024-03-26T04:31:10.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:10 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-120", "mode": "writeback"}]: dispatch 2024-03-26T04:31:10.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:10 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-120", "mode": "writeback"}]: dispatch 2024-03-26T04:31:10.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:10 smithi016 ceph-mon[37215]: pgmap v941: 268 pgs: 4 creating+activating, 24 creating+peering, 240 active+clean; 582 KiB data, 615 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:31:10.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:10 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-120"}]': finished 2024-03-26T04:31:10.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:10 smithi016 ceph-mon[37215]: osdmap e610: 8 total, 8 up, 8 in 2024-03-26T04:31:10.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:10 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-120", "mode": "writeback"}]: dispatch 2024-03-26T04:31:10.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:10 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-120", "mode": "writeback"}]: dispatch 2024-03-26T04:31:11.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:11 smithi183 ceph-mon[33745]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:31:11.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:11 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-120", "mode": "writeback"}]': finished 2024-03-26T04:31:11.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:11 smithi183 ceph-mon[33745]: osdmap e611: 8 total, 8 up, 8 in 2024-03-26T04:31:11.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:11 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:31:11.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:11 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:31:11.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:11 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:31:11.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:11 smithi183 ceph-mon[33745]: osdmap e612: 8 total, 8 up, 8 in 2024-03-26T04:31:11.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:11 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-120"}]: dispatch 2024-03-26T04:31:11.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:11 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-120"}]: dispatch 2024-03-26T04:31:11.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:11 smithi016 ceph-mon[29101]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:31:11.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:11 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-120", "mode": "writeback"}]': finished 2024-03-26T04:31:11.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:11 smithi016 ceph-mon[29101]: osdmap e611: 8 total, 8 up, 8 in 2024-03-26T04:31:11.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:11 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:31:11.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:11 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:31:11.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:11 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:31:11.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:11 smithi016 ceph-mon[29101]: osdmap e612: 8 total, 8 up, 8 in 2024-03-26T04:31:11.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:11 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-120"}]: dispatch 2024-03-26T04:31:11.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:11 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-120"}]: dispatch 2024-03-26T04:31:11.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:11 smithi016 ceph-mon[37215]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:31:11.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:11 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-120", "mode": "writeback"}]': finished 2024-03-26T04:31:11.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:11 smithi016 ceph-mon[37215]: osdmap e611: 8 total, 8 up, 8 in 2024-03-26T04:31:11.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:11 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:31:11.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:11 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:31:11.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:11 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:31:11.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:11 smithi016 ceph-mon[37215]: osdmap e612: 8 total, 8 up, 8 in 2024-03-26T04:31:11.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:11 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-120"}]: dispatch 2024-03-26T04:31:11.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:11 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-120"}]: dispatch 2024-03-26T04:31:12.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:12 smithi183 ceph-mon[33745]: pgmap v945: 268 pgs: 4 creating+activating, 24 creating+peering, 240 active+clean; 582 KiB data, 615 MiB used, 715 GiB / 715 GiB avail; 256 B/s wr, 0 op/s 2024-03-26T04:31:12.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:12 smithi183 ceph-mon[33745]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:31:12.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:12 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-120"}]': finished 2024-03-26T04:31:12.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:12 smithi183 ceph-mon[33745]: osdmap e613: 8 total, 8 up, 8 in 2024-03-26T04:31:12.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:12 smithi016 ceph-mon[29101]: pgmap v945: 268 pgs: 4 creating+activating, 24 creating+peering, 240 active+clean; 582 KiB data, 615 MiB used, 715 GiB / 715 GiB avail; 256 B/s wr, 0 op/s 2024-03-26T04:31:12.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:12 smithi016 ceph-mon[29101]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:31:12.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:12 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-120"}]': finished 2024-03-26T04:31:12.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:12 smithi016 ceph-mon[29101]: osdmap e613: 8 total, 8 up, 8 in 2024-03-26T04:31:12.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:12 smithi016 ceph-mon[37215]: pgmap v945: 268 pgs: 4 creating+activating, 24 creating+peering, 240 active+clean; 582 KiB data, 615 MiB used, 715 GiB / 715 GiB avail; 256 B/s wr, 0 op/s 2024-03-26T04:31:12.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:12 smithi016 ceph-mon[37215]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:31:12.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:12 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-120"}]': finished 2024-03-26T04:31:12.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:12 smithi016 ceph-mon[37215]: osdmap e613: 8 total, 8 up, 8 in 2024-03-26T04:31:13.041 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:31:12 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:31:13.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:13 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:31:13.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:13 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:31:13.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:13 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:31:14.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:14 smithi183 ceph-mon[33745]: pgmap v947: 268 pgs: 268 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-26T04:31:14.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:14 smithi183 ceph-mon[33745]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:31:14.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:14 smithi183 ceph-mon[33745]: osdmap e614: 8 total, 8 up, 8 in 2024-03-26T04:31:14.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:14 smithi016 ceph-mon[29101]: pgmap v947: 268 pgs: 268 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-26T04:31:14.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:14 smithi016 ceph-mon[29101]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:31:14.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:14 smithi016 ceph-mon[29101]: osdmap e614: 8 total, 8 up, 8 in 2024-03-26T04:31:14.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:14 smithi016 ceph-mon[37215]: pgmap v947: 268 pgs: 268 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-26T04:31:14.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:14 smithi016 ceph-mon[37215]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:31:14.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:14 smithi016 ceph-mon[37215]: osdmap e614: 8 total, 8 up, 8 in 2024-03-26T04:31:15.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:15 smithi183 ceph-mon[33745]: osdmap e615: 8 total, 8 up, 8 in 2024-03-26T04:31:15.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:15 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:31:15.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:15 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:31:15.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:15 smithi016 ceph-mon[29101]: osdmap e615: 8 total, 8 up, 8 in 2024-03-26T04:31:15.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:15 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:31:15.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:15 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:31:15.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:15 smithi016 ceph-mon[37215]: osdmap e615: 8 total, 8 up, 8 in 2024-03-26T04:31:15.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:15 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:31:15.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:15 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-122","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:31:16.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:16 smithi183 ceph-mon[33745]: pgmap v950: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 254 B/s wr, 1 op/s 2024-03-26T04:31:16.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:16 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:31:16.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:16 smithi183 ceph-mon[33745]: osdmap e616: 8 total, 8 up, 8 in 2024-03-26T04:31:16.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:16 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:31:16.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:16 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:31:16.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:16 smithi016 ceph-mon[29101]: pgmap v950: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 254 B/s wr, 1 op/s 2024-03-26T04:31:16.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:16 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:31:16.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:16 smithi016 ceph-mon[29101]: osdmap e616: 8 total, 8 up, 8 in 2024-03-26T04:31:16.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:16 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:31:16.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:16 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:31:16.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:16 smithi016 ceph-mon[37215]: pgmap v950: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 620 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 254 B/s wr, 1 op/s 2024-03-26T04:31:16.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:16 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-122","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:31:16.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:16 smithi016 ceph-mon[37215]: osdmap e616: 8 total, 8 up, 8 in 2024-03-26T04:31:16.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:16 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:31:16.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:16 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-122", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:31:17.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:17 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-122", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:31:17.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:17 smithi183 ceph-mon[33745]: osdmap e617: 8 total, 8 up, 8 in 2024-03-26T04:31:17.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:17 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-122"}]: dispatch 2024-03-26T04:31:17.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:17 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-122"}]: dispatch 2024-03-26T04:31:17.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:17 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-122", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:31:17.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:17 smithi016 ceph-mon[29101]: osdmap e617: 8 total, 8 up, 8 in 2024-03-26T04:31:17.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:17 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-122"}]: dispatch 2024-03-26T04:31:17.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:17 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-122"}]: dispatch 2024-03-26T04:31:17.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:17 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-122", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:31:17.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:17 smithi016 ceph-mon[37215]: osdmap e617: 8 total, 8 up, 8 in 2024-03-26T04:31:17.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:17 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-122"}]: dispatch 2024-03-26T04:31:17.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:17 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-122"}]: dispatch 2024-03-26T04:31:18.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:18 smithi183 ceph-mon[33745]: pgmap v953: 268 pgs: 13 unknown, 255 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:31:18.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:18 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-122"}]': finished 2024-03-26T04:31:18.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:18 smithi183 ceph-mon[33745]: osdmap e618: 8 total, 8 up, 8 in 2024-03-26T04:31:18.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:18 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-122", "mode": "writeback"}]: dispatch 2024-03-26T04:31:18.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:18 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-122", "mode": "writeback"}]: dispatch 2024-03-26T04:31:18.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:18 smithi016 ceph-mon[29101]: pgmap v953: 268 pgs: 13 unknown, 255 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:31:18.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:18 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-122"}]': finished 2024-03-26T04:31:18.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:18 smithi016 ceph-mon[29101]: osdmap e618: 8 total, 8 up, 8 in 2024-03-26T04:31:18.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:18 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-122", "mode": "writeback"}]: dispatch 2024-03-26T04:31:18.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:18 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-122", "mode": "writeback"}]: dispatch 2024-03-26T04:31:18.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:18 smithi016 ceph-mon[37215]: pgmap v953: 268 pgs: 13 unknown, 255 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:31:18.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:18 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-122"}]': finished 2024-03-26T04:31:18.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:18 smithi016 ceph-mon[37215]: osdmap e618: 8 total, 8 up, 8 in 2024-03-26T04:31:18.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:18 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-122", "mode": "writeback"}]: dispatch 2024-03-26T04:31:18.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:18 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-122", "mode": "writeback"}]: dispatch 2024-03-26T04:31:19.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:31:18 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:31:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:31:19.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:19 smithi183 ceph-mon[33745]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:31:19.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:19 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-122", "mode": "writeback"}]': finished 2024-03-26T04:31:19.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:19 smithi183 ceph-mon[33745]: osdmap e619: 8 total, 8 up, 8 in 2024-03-26T04:31:19.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:19 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:31:19.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:19 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:31:19.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:19 smithi016 ceph-mon[37215]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:31:19.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:19 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-122", "mode": "writeback"}]': finished 2024-03-26T04:31:19.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:19 smithi016 ceph-mon[37215]: osdmap e619: 8 total, 8 up, 8 in 2024-03-26T04:31:19.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:19 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:31:19.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:19 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:31:19.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:19 smithi016 ceph-mon[29101]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:31:19.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:19 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-122", "mode": "writeback"}]': finished 2024-03-26T04:31:19.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:19 smithi016 ceph-mon[29101]: osdmap e619: 8 total, 8 up, 8 in 2024-03-26T04:31:19.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:19 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:31:19.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:19 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:31:20.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:20 smithi016 ceph-mon[37215]: pgmap v956: 268 pgs: 268 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:31:20.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:20 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:31:20.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:20 smithi016 ceph-mon[37215]: osdmap e620: 8 total, 8 up, 8 in 2024-03-26T04:31:20.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:20 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-122"}]: dispatch 2024-03-26T04:31:20.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:20 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-122"}]: dispatch 2024-03-26T04:31:20.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:20 smithi016 ceph-mon[29101]: pgmap v956: 268 pgs: 268 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:31:20.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:20 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:31:20.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:20 smithi016 ceph-mon[29101]: osdmap e620: 8 total, 8 up, 8 in 2024-03-26T04:31:20.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:20 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-122"}]: dispatch 2024-03-26T04:31:20.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:20 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-122"}]: dispatch 2024-03-26T04:31:20.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:20 smithi183 ceph-mon[33745]: pgmap v956: 268 pgs: 268 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:31:20.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:20 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:31:20.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:20 smithi183 ceph-mon[33745]: osdmap e620: 8 total, 8 up, 8 in 2024-03-26T04:31:20.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:20 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-122"}]: dispatch 2024-03-26T04:31:20.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:20 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-122"}]: dispatch 2024-03-26T04:31:21.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:21 smithi016 ceph-mon[37215]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:31:21.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:21 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-122"}]': finished 2024-03-26T04:31:21.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:21 smithi016 ceph-mon[37215]: osdmap e621: 8 total, 8 up, 8 in 2024-03-26T04:31:21.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:21 smithi016 ceph-mon[37215]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:31:21.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:21 smithi016 ceph-mon[29101]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:31:21.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:21 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-122"}]': finished 2024-03-26T04:31:21.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:21 smithi016 ceph-mon[29101]: osdmap e621: 8 total, 8 up, 8 in 2024-03-26T04:31:21.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:21 smithi016 ceph-mon[29101]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:31:21.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:21 smithi183 ceph-mon[33745]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:31:21.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:21 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-122"}]': finished 2024-03-26T04:31:21.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:21 smithi183 ceph-mon[33745]: osdmap e621: 8 total, 8 up, 8 in 2024-03-26T04:31:21.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:21 smithi183 ceph-mon[33745]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:31:22.644 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:22 smithi183 ceph-mon[33745]: pgmap v959: 268 pgs: 268 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-03-26T04:31:22.644 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:22 smithi183 ceph-mon[33745]: osdmap e622: 8 total, 8 up, 8 in 2024-03-26T04:31:22.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:22 smithi016 ceph-mon[37215]: pgmap v959: 268 pgs: 268 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-03-26T04:31:22.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:22 smithi016 ceph-mon[37215]: osdmap e622: 8 total, 8 up, 8 in 2024-03-26T04:31:22.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:22 smithi016 ceph-mon[29101]: pgmap v959: 268 pgs: 268 active+clean; 582 KiB data, 621 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 0 op/s 2024-03-26T04:31:22.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:22 smithi016 ceph-mon[29101]: osdmap e622: 8 total, 8 up, 8 in 2024-03-26T04:31:23.041 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:31:22 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:31:23.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:23 smithi016 ceph-mon[37215]: osdmap e623: 8 total, 8 up, 8 in 2024-03-26T04:31:23.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:23 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:31:23.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:23 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:31:23.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:23 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:31:23.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:23 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:31:23.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:23 smithi016 ceph-mon[29101]: osdmap e623: 8 total, 8 up, 8 in 2024-03-26T04:31:23.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:23 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:31:23.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:23 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:31:23.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:23 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:31:23.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:23 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:31:23.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:23 smithi183 ceph-mon[33745]: osdmap e623: 8 total, 8 up, 8 in 2024-03-26T04:31:23.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:23 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:31:23.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:23 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-124","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:31:23.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:23 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:31:23.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:23 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:31:24.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:24 smithi016 ceph-mon[29101]: pgmap v962: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:31:24.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:24 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:31:24.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:24 smithi016 ceph-mon[29101]: osdmap e624: 8 total, 8 up, 8 in 2024-03-26T04:31:24.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:24 smithi016 ceph-mon[37215]: pgmap v962: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:31:24.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:24 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:31:24.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:24 smithi016 ceph-mon[37215]: osdmap e624: 8 total, 8 up, 8 in 2024-03-26T04:31:24.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:24 smithi183 ceph-mon[33745]: pgmap v962: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:31:24.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:24 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-124","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:31:24.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:24 smithi183 ceph-mon[33745]: osdmap e624: 8 total, 8 up, 8 in 2024-03-26T04:31:25.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:25 smithi016 ceph-mon[29101]: osdmap e625: 8 total, 8 up, 8 in 2024-03-26T04:31:25.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:25 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:31:25.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:25 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:31:25.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:25 smithi016 ceph-mon[37215]: osdmap e625: 8 total, 8 up, 8 in 2024-03-26T04:31:25.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:25 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:31:25.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:25 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:31:25.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:25 smithi183 ceph-mon[33745]: osdmap e625: 8 total, 8 up, 8 in 2024-03-26T04:31:25.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:25 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:31:25.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:25 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-124", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:31:26.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:26 smithi016 ceph-mon[29101]: pgmap v965: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:31:26.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:26 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-124", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:31:26.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:26 smithi016 ceph-mon[29101]: osdmap e626: 8 total, 8 up, 8 in 2024-03-26T04:31:26.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:26 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-124"}]: dispatch 2024-03-26T04:31:26.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:26 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-124"}]: dispatch 2024-03-26T04:31:26.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:26 smithi016 ceph-mon[37215]: pgmap v965: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:31:26.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:26 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-124", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:31:26.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:26 smithi016 ceph-mon[37215]: osdmap e626: 8 total, 8 up, 8 in 2024-03-26T04:31:26.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:26 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-124"}]: dispatch 2024-03-26T04:31:26.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:26 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-124"}]: dispatch 2024-03-26T04:31:26.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:26 smithi183 ceph-mon[33745]: pgmap v965: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:31:26.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:26 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-124", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:31:26.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:26 smithi183 ceph-mon[33745]: osdmap e626: 8 total, 8 up, 8 in 2024-03-26T04:31:26.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:26 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-124"}]: dispatch 2024-03-26T04:31:26.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:26 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-124"}]: dispatch 2024-03-26T04:31:27.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:27 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-124"}]': finished 2024-03-26T04:31:27.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:27 smithi016 ceph-mon[37215]: osdmap e627: 8 total, 8 up, 8 in 2024-03-26T04:31:27.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:27 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-124", "mode": "writeback"}]: dispatch 2024-03-26T04:31:27.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:27 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-124", "mode": "writeback"}]: dispatch 2024-03-26T04:31:27.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:27 smithi016 ceph-mon[37215]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:31:27.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:27 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-124", "mode": "writeback"}]': finished 2024-03-26T04:31:27.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:27 smithi016 ceph-mon[37215]: osdmap e628: 8 total, 8 up, 8 in 2024-03-26T04:31:27.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:27 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-124"}]': finished 2024-03-26T04:31:27.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:27 smithi016 ceph-mon[29101]: osdmap e627: 8 total, 8 up, 8 in 2024-03-26T04:31:27.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:27 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-124", "mode": "writeback"}]: dispatch 2024-03-26T04:31:27.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:27 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-124", "mode": "writeback"}]: dispatch 2024-03-26T04:31:27.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:27 smithi016 ceph-mon[29101]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:31:27.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:27 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-124", "mode": "writeback"}]': finished 2024-03-26T04:31:27.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:27 smithi016 ceph-mon[29101]: osdmap e628: 8 total, 8 up, 8 in 2024-03-26T04:31:27.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:27 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-124"}]': finished 2024-03-26T04:31:27.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:27 smithi183 ceph-mon[33745]: osdmap e627: 8 total, 8 up, 8 in 2024-03-26T04:31:27.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:27 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-124", "mode": "writeback"}]: dispatch 2024-03-26T04:31:27.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:27 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-124", "mode": "writeback"}]: dispatch 2024-03-26T04:31:27.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:27 smithi183 ceph-mon[33745]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:31:27.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:27 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-124", "mode": "writeback"}]': finished 2024-03-26T04:31:27.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:27 smithi183 ceph-mon[33745]: osdmap e628: 8 total, 8 up, 8 in 2024-03-26T04:31:28.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:28 smithi016 ceph-mon[37215]: pgmap v968: 268 pgs: 13 unknown, 255 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:31:28.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:28 smithi016 ceph-mon[29101]: pgmap v968: 268 pgs: 13 unknown, 255 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:31:28.692 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:28 smithi183 ceph-mon[33745]: pgmap v968: 268 pgs: 13 unknown, 255 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-26T04:31:29.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:31:28 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:31:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:31:29.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:29 smithi016 ceph-mon[29101]: osdmap e629: 8 total, 8 up, 8 in 2024-03-26T04:31:29.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:29 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:31:29.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:29 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:31:29.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:29 smithi016 ceph-mon[37215]: osdmap e629: 8 total, 8 up, 8 in 2024-03-26T04:31:29.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:29 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:31:29.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:29 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:31:29.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:29 smithi183 ceph-mon[33745]: osdmap e629: 8 total, 8 up, 8 in 2024-03-26T04:31:29.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:29 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:31:29.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:29 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:31:30.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:30 smithi016 ceph-mon[29101]: pgmap v971: 268 pgs: 268 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T04:31:30.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:30 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:31:30.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:30 smithi016 ceph-mon[29101]: osdmap e630: 8 total, 8 up, 8 in 2024-03-26T04:31:30.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:30 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-124"}]: dispatch 2024-03-26T04:31:30.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:30 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-124"}]: dispatch 2024-03-26T04:31:30.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:30 smithi016 ceph-mon[37215]: pgmap v971: 268 pgs: 268 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T04:31:30.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:30 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:31:30.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:30 smithi016 ceph-mon[37215]: osdmap e630: 8 total, 8 up, 8 in 2024-03-26T04:31:30.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:30 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-124"}]: dispatch 2024-03-26T04:31:30.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:30 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-124"}]: dispatch 2024-03-26T04:31:30.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:30 smithi183 ceph-mon[33745]: pgmap v971: 268 pgs: 268 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.7 KiB/s wr, 3 op/s 2024-03-26T04:31:30.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:30 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:31:30.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:30 smithi183 ceph-mon[33745]: osdmap e630: 8 total, 8 up, 8 in 2024-03-26T04:31:30.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:30 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-124"}]: dispatch 2024-03-26T04:31:30.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:30 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-124"}]: dispatch 2024-03-26T04:31:31.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:31 smithi016 ceph-mon[29101]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:31:31.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:31 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-124"}]': finished 2024-03-26T04:31:31.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:31 smithi016 ceph-mon[29101]: osdmap e631: 8 total, 8 up, 8 in 2024-03-26T04:31:31.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:31 smithi016 ceph-mon[37215]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:31:31.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:31 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-124"}]': finished 2024-03-26T04:31:31.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:31 smithi016 ceph-mon[37215]: osdmap e631: 8 total, 8 up, 8 in 2024-03-26T04:31:31.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:31 smithi183 ceph-mon[33745]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:31:31.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:31 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-124"}]': finished 2024-03-26T04:31:31.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:31 smithi183 ceph-mon[33745]: osdmap e631: 8 total, 8 up, 8 in 2024-03-26T04:31:32.655 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:32 smithi183 ceph-mon[33745]: pgmap v974: 268 pgs: 268 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1023 B/s wr, 0 op/s 2024-03-26T04:31:32.655 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:32 smithi183 ceph-mon[33745]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:31:32.655 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:32 smithi183 ceph-mon[33745]: osdmap e632: 8 total, 8 up, 8 in 2024-03-26T04:31:32.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:32 smithi016 ceph-mon[37215]: pgmap v974: 268 pgs: 268 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1023 B/s wr, 0 op/s 2024-03-26T04:31:32.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:32 smithi016 ceph-mon[37215]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:31:32.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:32 smithi016 ceph-mon[37215]: osdmap e632: 8 total, 8 up, 8 in 2024-03-26T04:31:32.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:32 smithi016 ceph-mon[29101]: pgmap v974: 268 pgs: 268 active+clean; 582 KiB data, 626 MiB used, 715 GiB / 715 GiB avail; 1023 B/s wr, 0 op/s 2024-03-26T04:31:32.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:32 smithi016 ceph-mon[29101]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:31:32.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:32 smithi016 ceph-mon[29101]: osdmap e632: 8 total, 8 up, 8 in 2024-03-26T04:31:33.041 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:31:32 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:31:33.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:33 smithi016 ceph-mon[37215]: osdmap e633: 8 total, 8 up, 8 in 2024-03-26T04:31:33.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:33 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:31:33.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:33 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:31:33.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:33 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:31:33.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:33 smithi016 ceph-mon[29101]: osdmap e633: 8 total, 8 up, 8 in 2024-03-26T04:31:33.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:33 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:31:33.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:33 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:31:33.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:33 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:31:33.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:33 smithi183 ceph-mon[33745]: osdmap e633: 8 total, 8 up, 8 in 2024-03-26T04:31:33.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:33 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:31:33.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:33 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-126","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:31:33.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:33 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:31:34.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:34 smithi016 ceph-mon[37215]: pgmap v977: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:31:34.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:34 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:31:34.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:34 smithi016 ceph-mon[37215]: osdmap e634: 8 total, 8 up, 8 in 2024-03-26T04:31:34.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:34 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:31:34.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:34 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:31:34.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:34 smithi016 ceph-mon[29101]: pgmap v977: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:31:34.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:34 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:31:34.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:34 smithi016 ceph-mon[29101]: osdmap e634: 8 total, 8 up, 8 in 2024-03-26T04:31:34.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:34 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:31:34.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:34 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:31:34.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:34 smithi183 ceph-mon[33745]: pgmap v977: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:31:34.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:34 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-126","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:31:34.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:34 smithi183 ceph-mon[33745]: osdmap e634: 8 total, 8 up, 8 in 2024-03-26T04:31:34.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:34 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:31:34.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:34 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-126", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:31:35.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:35 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-126", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:31:35.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:35 smithi016 ceph-mon[37215]: osdmap e635: 8 total, 8 up, 8 in 2024-03-26T04:31:35.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:35 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-126"}]: dispatch 2024-03-26T04:31:35.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:35 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-126"}]: dispatch 2024-03-26T04:31:35.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:35 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-126"}]': finished 2024-03-26T04:31:35.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:35 smithi016 ceph-mon[37215]: osdmap e636: 8 total, 8 up, 8 in 2024-03-26T04:31:35.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:35 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-126", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:31:35.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:35 smithi016 ceph-mon[29101]: osdmap e635: 8 total, 8 up, 8 in 2024-03-26T04:31:35.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:35 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-126"}]: dispatch 2024-03-26T04:31:35.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:35 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-126"}]: dispatch 2024-03-26T04:31:35.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:35 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-126"}]': finished 2024-03-26T04:31:35.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:35 smithi016 ceph-mon[29101]: osdmap e636: 8 total, 8 up, 8 in 2024-03-26T04:31:35.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:35 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-126", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:31:35.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:35 smithi183 ceph-mon[33745]: osdmap e635: 8 total, 8 up, 8 in 2024-03-26T04:31:35.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:35 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-126"}]: dispatch 2024-03-26T04:31:35.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:35 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-126"}]: dispatch 2024-03-26T04:31:35.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:35 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-126"}]': finished 2024-03-26T04:31:35.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:35 smithi183 ceph-mon[33745]: osdmap e636: 8 total, 8 up, 8 in 2024-03-26T04:31:36.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:36 smithi016 ceph-mon[29101]: pgmap v980: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:31:36.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:36 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-126", "mode": "writeback"}]: dispatch 2024-03-26T04:31:36.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:36 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-126", "mode": "writeback"}]: dispatch 2024-03-26T04:31:36.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:36 smithi016 ceph-mon[29101]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:31:36.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:36 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-126", "mode": "writeback"}]': finished 2024-03-26T04:31:36.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:36 smithi016 ceph-mon[29101]: osdmap e637: 8 total, 8 up, 8 in 2024-03-26T04:31:36.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:36 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:31:36.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:36 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:31:36.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:36 smithi016 ceph-mon[37215]: pgmap v980: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:31:36.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:36 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-126", "mode": "writeback"}]: dispatch 2024-03-26T04:31:36.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:36 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-126", "mode": "writeback"}]: dispatch 2024-03-26T04:31:36.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:36 smithi016 ceph-mon[37215]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:31:36.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:36 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-126", "mode": "writeback"}]': finished 2024-03-26T04:31:36.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:36 smithi016 ceph-mon[37215]: osdmap e637: 8 total, 8 up, 8 in 2024-03-26T04:31:36.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:36 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:31:36.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:36 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:31:36.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:36 smithi183 ceph-mon[33745]: pgmap v980: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:31:36.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:36 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-126", "mode": "writeback"}]: dispatch 2024-03-26T04:31:36.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:36 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-126", "mode": "writeback"}]: dispatch 2024-03-26T04:31:36.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:36 smithi183 ceph-mon[33745]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:31:36.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:36 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-126", "mode": "writeback"}]': finished 2024-03-26T04:31:36.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:36 smithi183 ceph-mon[33745]: osdmap e637: 8 total, 8 up, 8 in 2024-03-26T04:31:36.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:36 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:31:36.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:36 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:31:38.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:38 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:31:38.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:38 smithi183 ceph-mon[33745]: osdmap e638: 8 total, 8 up, 8 in 2024-03-26T04:31:38.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:38 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-126"}]: dispatch 2024-03-26T04:31:38.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:38 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-126"}]: dispatch 2024-03-26T04:31:38.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:38 smithi183 ceph-mon[33745]: pgmap v984: 268 pgs: 11 unknown, 257 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T04:31:38.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:38 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:31:38.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:38 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:31:38.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:38 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:31:38.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:38 smithi016 ceph-mon[37215]: osdmap e638: 8 total, 8 up, 8 in 2024-03-26T04:31:38.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:38 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-126"}]: dispatch 2024-03-26T04:31:38.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:38 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-126"}]: dispatch 2024-03-26T04:31:38.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:38 smithi016 ceph-mon[37215]: pgmap v984: 268 pgs: 11 unknown, 257 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T04:31:38.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:38 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:31:38.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:38 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:31:38.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:38 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:31:38.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:38 smithi016 ceph-mon[29101]: osdmap e638: 8 total, 8 up, 8 in 2024-03-26T04:31:38.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:38 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-126"}]: dispatch 2024-03-26T04:31:38.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:38 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-126"}]: dispatch 2024-03-26T04:31:38.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:38 smithi016 ceph-mon[29101]: pgmap v984: 268 pgs: 11 unknown, 257 active+clean; 582 KiB data, 627 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T04:31:38.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:38 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:31:38.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:38 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:31:39.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:31:38 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:31:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:31:39.216 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T04:31:39.216 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (9048 ms) 2024-03-26T04:31:39.216 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-03-26T04:31:39.216 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T04:31:39.216 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (6037 ms) 2024-03-26T04:31:39.216 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-03-26T04:31:39.216 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T04:31:39.216 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (5013 ms) 2024-03-26T04:31:39.216 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-03-26T04:31:39.216 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T04:31:39.216 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (25434 ms) 2024-03-26T04:31:39.217 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-03-26T04:31:39.217 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T04:31:39.217 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (14050 ms) 2024-03-26T04:31:39.217 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-03-26T04:31:39.217 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (12080 ms) 2024-03-26T04:31:39.217 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-03-26T04:31:39.217 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-03-26T04:31:39.217 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: 00000010 69 73 |is| 2024-03-26T04:31:39.217 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: 00000012 2024-03-26T04:31:39.217 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (12062 ms) 2024-03-26T04:31:39.217 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-03-26T04:31:39.217 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: require_osd_release = squid 2024-03-26T04:31:39.217 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (6042 ms) 2024-03-26T04:31:39.217 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (553479 ms total) 2024-03-26T04:31:39.217 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: 2024-03-26T04:31:39.218 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-03-26T04:31:39.218 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-03-26T04:31:39.218 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1020 ms) 2024-03-26T04:31:39.218 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-03-26T04:31:39.218 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (11065 ms) 2024-03-26T04:31:39.218 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-03-26T04:31:39.218 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (18072 ms) 2024-03-26T04:31:39.218 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-03-26T04:31:39.218 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (2 ms) 2024-03-26T04:31:39.218 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (30159 ms total) 2024-03-26T04:31:39.218 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: 2024-03-26T04:31:39.218 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-03-26T04:31:39.218 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-03-26T04:31:39.219 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (7038 ms) 2024-03-26T04:31:39.219 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-03-26T04:31:39.219 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (8039 ms) 2024-03-26T04:31:39.219 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-03-26T04:31:39.219 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: waiting for scrub... 2024-03-26T04:31:39.219 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: done waiting 2024-03-26T04:31:39.219 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (24476 ms) 2024-03-26T04:31:39.219 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-03-26T04:31:39.219 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (9922 ms) 2024-03-26T04:31:39.219 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-03-26T04:31:39.219 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (8038 ms) 2024-03-26T04:31:39.219 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-03-26T04:31:39.219 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (8052 ms) 2024-03-26T04:31:39.219 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-03-26T04:31:39.219 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (10072 ms) 2024-03-26T04:31:39.220 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-03-26T04:31:39.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:39 smithi183 ceph-mon[33745]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:31:39.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:39 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-126"}]': finished 2024-03-26T04:31:39.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:39 smithi183 ceph-mon[33745]: osdmap e639: 8 total, 8 up, 8 in 2024-03-26T04:31:39.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:39 smithi016 ceph-mon[37215]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:31:39.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:39 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-126"}]': finished 2024-03-26T04:31:39.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:39 smithi016 ceph-mon[37215]: osdmap e639: 8 total, 8 up, 8 in 2024-03-26T04:31:39.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:39 smithi016 ceph-mon[29101]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:31:39.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:39 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-126"}]': finished 2024-03-26T04:31:39.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:39 smithi016 ceph-mon[29101]: osdmap e639: 8 total, 8 up, 8 in 2024-03-26T04:31:40.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:40 smithi183 ceph-mon[33745]: osdmap e640: 8 total, 8 up, 8 in 2024-03-26T04:31:40.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:40 smithi183 ceph-mon[33745]: pgmap v987: 236 pgs: 236 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 267 B/s wr, 1 op/s 2024-03-26T04:31:40.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:40 smithi016 ceph-mon[37215]: osdmap e640: 8 total, 8 up, 8 in 2024-03-26T04:31:40.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:40 smithi016 ceph-mon[37215]: pgmap v987: 236 pgs: 236 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 267 B/s wr, 1 op/s 2024-03-26T04:31:40.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:40 smithi016 ceph-mon[29101]: osdmap e640: 8 total, 8 up, 8 in 2024-03-26T04:31:40.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:40 smithi016 ceph-mon[29101]: pgmap v987: 236 pgs: 236 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 267 B/s wr, 1 op/s 2024-03-26T04:31:41.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:41 smithi183 ceph-mon[33745]: osdmap e641: 8 total, 8 up, 8 in 2024-03-26T04:31:41.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:41 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:31:41.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:41 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:31:41.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:41 smithi183 ceph-mon[33745]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:31:41.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:41 smithi016 ceph-mon[29101]: osdmap e641: 8 total, 8 up, 8 in 2024-03-26T04:31:41.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:41 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:31:41.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:41 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:31:41.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:41 smithi016 ceph-mon[29101]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:31:41.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:41 smithi016 ceph-mon[37215]: osdmap e641: 8 total, 8 up, 8 in 2024-03-26T04:31:41.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:41 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:31:41.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:41 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-128","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:31:41.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:41 smithi016 ceph-mon[37215]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:31:42.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:42 smithi183 ceph-mon[33745]: pgmap v989: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 254 B/s wr, 0 op/s 2024-03-26T04:31:42.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:42 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:31:42.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:42 smithi183 ceph-mon[33745]: osdmap e642: 8 total, 8 up, 8 in 2024-03-26T04:31:42.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:42 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:31:42.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:42 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:31:42.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:42 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:31:42.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:42 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:31:42.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:42 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:31:42.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:42 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:31:42.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:42 smithi016 ceph-mon[37215]: pgmap v989: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 254 B/s wr, 0 op/s 2024-03-26T04:31:42.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:42 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:31:42.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:42 smithi016 ceph-mon[37215]: osdmap e642: 8 total, 8 up, 8 in 2024-03-26T04:31:42.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:42 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:31:42.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:42 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:31:42.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:42 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:31:42.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:42 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:31:42.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:42 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:31:42.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:42 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:31:42.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:42 smithi016 ceph-mon[29101]: pgmap v989: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 628 MiB used, 715 GiB / 715 GiB avail; 254 B/s wr, 0 op/s 2024-03-26T04:31:42.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:42 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-128","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:31:42.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:42 smithi016 ceph-mon[29101]: osdmap e642: 8 total, 8 up, 8 in 2024-03-26T04:31:42.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:42 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:31:42.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:42 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-128", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:31:42.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:42 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:31:42.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:42 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:31:42.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:42 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:31:42.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:42 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:31:43.042 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:31:42 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:31:43.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:43 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-128", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:31:43.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:43 smithi183 ceph-mon[33745]: osdmap e643: 8 total, 8 up, 8 in 2024-03-26T04:31:43.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:43 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-128"}]: dispatch 2024-03-26T04:31:43.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:43 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-128"}]: dispatch 2024-03-26T04:31:43.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:43 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:31:43.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:43 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-128"}]': finished 2024-03-26T04:31:43.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:43 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-128", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:31:43.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:43 smithi016 ceph-mon[37215]: osdmap e643: 8 total, 8 up, 8 in 2024-03-26T04:31:43.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:43 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-128"}]: dispatch 2024-03-26T04:31:43.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:43 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-128"}]: dispatch 2024-03-26T04:31:43.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:43 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:31:43.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:43 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-128"}]': finished 2024-03-26T04:31:43.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:43 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-128", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:31:43.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:43 smithi016 ceph-mon[29101]: osdmap e643: 8 total, 8 up, 8 in 2024-03-26T04:31:43.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:43 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-128"}]: dispatch 2024-03-26T04:31:43.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:43 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-128"}]: dispatch 2024-03-26T04:31:43.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:43 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:31:43.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:43 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-128"}]': finished 2024-03-26T04:31:44.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:44 smithi183 ceph-mon[33745]: pgmap v992: 268 pgs: 268 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:31:44.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:44 smithi183 ceph-mon[33745]: osdmap e644: 8 total, 8 up, 8 in 2024-03-26T04:31:44.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:44 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-128", "mode": "writeback"}]: dispatch 2024-03-26T04:31:44.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:44 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-128", "mode": "writeback"}]: dispatch 2024-03-26T04:31:44.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:44 smithi183 ceph-mon[33745]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:31:44.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:44 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-128", "mode": "writeback"}]': finished 2024-03-26T04:31:44.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:44 smithi183 ceph-mon[33745]: osdmap e645: 8 total, 8 up, 8 in 2024-03-26T04:31:44.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:44 smithi016 ceph-mon[29101]: pgmap v992: 268 pgs: 268 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:31:44.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:44 smithi016 ceph-mon[29101]: osdmap e644: 8 total, 8 up, 8 in 2024-03-26T04:31:44.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:44 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-128", "mode": "writeback"}]: dispatch 2024-03-26T04:31:44.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:44 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-128", "mode": "writeback"}]: dispatch 2024-03-26T04:31:44.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:44 smithi016 ceph-mon[29101]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:31:44.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:44 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-128", "mode": "writeback"}]': finished 2024-03-26T04:31:44.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:44 smithi016 ceph-mon[29101]: osdmap e645: 8 total, 8 up, 8 in 2024-03-26T04:31:44.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:44 smithi016 ceph-mon[37215]: pgmap v992: 268 pgs: 268 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:31:44.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:44 smithi016 ceph-mon[37215]: osdmap e644: 8 total, 8 up, 8 in 2024-03-26T04:31:44.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:44 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-128", "mode": "writeback"}]: dispatch 2024-03-26T04:31:44.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:44 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-128", "mode": "writeback"}]: dispatch 2024-03-26T04:31:44.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:44 smithi016 ceph-mon[37215]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:31:44.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:44 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-128", "mode": "writeback"}]': finished 2024-03-26T04:31:44.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:44 smithi016 ceph-mon[37215]: osdmap e645: 8 total, 8 up, 8 in 2024-03-26T04:31:46.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:46 smithi183 ceph-mon[33745]: pgmap v995: 268 pgs: 268 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:31:46.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:46 smithi016 ceph-mon[29101]: pgmap v995: 268 pgs: 268 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:31:46.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:46 smithi016 ceph-mon[37215]: pgmap v995: 268 pgs: 268 active+clean; 582 KiB data, 636 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:31:48.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:48 smithi183 ceph-mon[33745]: pgmap v996: 268 pgs: 268 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T04:31:48.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:48 smithi016 ceph-mon[37215]: pgmap v996: 268 pgs: 268 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T04:31:48.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:48 smithi016 ceph-mon[29101]: pgmap v996: 268 pgs: 268 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T04:31:49.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:31:48 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:31:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:31:50.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:50 smithi183 ceph-mon[33745]: pgmap v997: 268 pgs: 268 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 878 B/s rd, 292 B/s wr, 1 op/s 2024-03-26T04:31:50.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:50 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:31:50.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:50 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:31:50.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:50 smithi016 ceph-mon[37215]: pgmap v997: 268 pgs: 268 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 878 B/s rd, 292 B/s wr, 1 op/s 2024-03-26T04:31:50.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:50 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:31:50.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:50 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:31:50.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:50 smithi016 ceph-mon[29101]: pgmap v997: 268 pgs: 268 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 878 B/s rd, 292 B/s wr, 1 op/s 2024-03-26T04:31:50.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:50 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:31:50.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:50 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:31:51.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:51 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:31:51.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:51 smithi183 ceph-mon[33745]: osdmap e646: 8 total, 8 up, 8 in 2024-03-26T04:31:51.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:51 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-128"}]: dispatch 2024-03-26T04:31:51.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:51 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-128"}]: dispatch 2024-03-26T04:31:51.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:51 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:31:51.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:51 smithi016 ceph-mon[37215]: osdmap e646: 8 total, 8 up, 8 in 2024-03-26T04:31:51.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:51 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-128"}]: dispatch 2024-03-26T04:31:51.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:51 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-128"}]: dispatch 2024-03-26T04:31:51.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:51 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:31:51.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:51 smithi016 ceph-mon[29101]: osdmap e646: 8 total, 8 up, 8 in 2024-03-26T04:31:51.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:51 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-128"}]: dispatch 2024-03-26T04:31:51.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:51 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-128"}]: dispatch 2024-03-26T04:31:52.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:52 smithi183 ceph-mon[33745]: pgmap v999: 268 pgs: 268 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 768 B/s rd, 256 B/s wr, 1 op/s 2024-03-26T04:31:52.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:52 smithi183 ceph-mon[33745]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:31:52.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:52 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-128"}]': finished 2024-03-26T04:31:52.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:52 smithi183 ceph-mon[33745]: osdmap e647: 8 total, 8 up, 8 in 2024-03-26T04:31:52.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:52 smithi016 ceph-mon[29101]: pgmap v999: 268 pgs: 268 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 768 B/s rd, 256 B/s wr, 1 op/s 2024-03-26T04:31:52.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:52 smithi016 ceph-mon[29101]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:31:52.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:52 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-128"}]': finished 2024-03-26T04:31:52.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:52 smithi016 ceph-mon[29101]: osdmap e647: 8 total, 8 up, 8 in 2024-03-26T04:31:52.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:52 smithi016 ceph-mon[37215]: pgmap v999: 268 pgs: 268 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 768 B/s rd, 256 B/s wr, 1 op/s 2024-03-26T04:31:52.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:52 smithi016 ceph-mon[37215]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:31:52.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:52 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-128"}]': finished 2024-03-26T04:31:52.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:52 smithi016 ceph-mon[37215]: osdmap e647: 8 total, 8 up, 8 in 2024-03-26T04:31:53.042 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:31:52 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:31:53.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:53 smithi183 ceph-mon[33745]: osdmap e648: 8 total, 8 up, 8 in 2024-03-26T04:31:53.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:53 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:31:53.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:53 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:31:53.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:53 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:31:53.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:53 smithi016 ceph-mon[29101]: osdmap e648: 8 total, 8 up, 8 in 2024-03-26T04:31:53.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:53 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:31:53.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:53 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:31:53.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:53 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:31:53.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:53 smithi016 ceph-mon[37215]: osdmap e648: 8 total, 8 up, 8 in 2024-03-26T04:31:53.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:53 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:31:53.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:53 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:31:53.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:53 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:31:54.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:54 smithi183 ceph-mon[33745]: pgmap v1002: 236 pgs: 236 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:31:54.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:54 smithi183 ceph-mon[33745]: osdmap e649: 8 total, 8 up, 8 in 2024-03-26T04:31:54.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:54 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:31:54.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:54 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:31:54.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:54 smithi016 ceph-mon[29101]: pgmap v1002: 236 pgs: 236 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:31:54.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:54 smithi016 ceph-mon[29101]: osdmap e649: 8 total, 8 up, 8 in 2024-03-26T04:31:54.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:54 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:31:54.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:54 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:31:54.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:54 smithi016 ceph-mon[37215]: pgmap v1002: 236 pgs: 236 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:31:54.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:54 smithi016 ceph-mon[37215]: osdmap e649: 8 total, 8 up, 8 in 2024-03-26T04:31:54.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:54 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:31:54.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:54 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-130","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:31:55.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:55 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:31:55.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:55 smithi183 ceph-mon[33745]: osdmap e650: 8 total, 8 up, 8 in 2024-03-26T04:31:55.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:55 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:31:55.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:55 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:31:55.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:55 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-130", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:31:55.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:55 smithi183 ceph-mon[33745]: osdmap e651: 8 total, 8 up, 8 in 2024-03-26T04:31:55.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:55 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-130"}]: dispatch 2024-03-26T04:31:55.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:55 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-130"}]: dispatch 2024-03-26T04:31:55.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:55 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:31:55.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:55 smithi016 ceph-mon[29101]: osdmap e650: 8 total, 8 up, 8 in 2024-03-26T04:31:55.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:55 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:31:55.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:55 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:31:55.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:55 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-130", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:31:55.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:55 smithi016 ceph-mon[29101]: osdmap e651: 8 total, 8 up, 8 in 2024-03-26T04:31:55.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:55 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-130"}]: dispatch 2024-03-26T04:31:55.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:55 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-130"}]: dispatch 2024-03-26T04:31:55.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:55 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-130","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:31:55.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:55 smithi016 ceph-mon[37215]: osdmap e650: 8 total, 8 up, 8 in 2024-03-26T04:31:55.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:55 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:31:55.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:55 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-130", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:31:55.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:55 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-130", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:31:55.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:55 smithi016 ceph-mon[37215]: osdmap e651: 8 total, 8 up, 8 in 2024-03-26T04:31:55.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:55 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-130"}]: dispatch 2024-03-26T04:31:55.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:55 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-130"}]: dispatch 2024-03-26T04:31:56.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:56 smithi183 ceph-mon[33745]: pgmap v1005: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:31:56.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:56 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-130"}]': finished 2024-03-26T04:31:56.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:56 smithi183 ceph-mon[33745]: osdmap e652: 8 total, 8 up, 8 in 2024-03-26T04:31:56.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:56 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-130", "mode": "writeback"}]: dispatch 2024-03-26T04:31:56.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:56 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-130", "mode": "writeback"}]: dispatch 2024-03-26T04:31:56.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:56 smithi016 ceph-mon[29101]: pgmap v1005: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:31:56.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:56 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-130"}]': finished 2024-03-26T04:31:56.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:56 smithi016 ceph-mon[29101]: osdmap e652: 8 total, 8 up, 8 in 2024-03-26T04:31:56.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:56 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-130", "mode": "writeback"}]: dispatch 2024-03-26T04:31:56.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:56 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-130", "mode": "writeback"}]: dispatch 2024-03-26T04:31:56.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:56 smithi016 ceph-mon[37215]: pgmap v1005: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:31:56.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:56 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-130"}]': finished 2024-03-26T04:31:56.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:56 smithi016 ceph-mon[37215]: osdmap e652: 8 total, 8 up, 8 in 2024-03-26T04:31:56.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:56 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-130", "mode": "writeback"}]: dispatch 2024-03-26T04:31:56.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:56 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-130", "mode": "writeback"}]: dispatch 2024-03-26T04:31:57.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:57 smithi183 ceph-mon[33745]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:31:57.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:57 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-130", "mode": "writeback"}]': finished 2024-03-26T04:31:57.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:57 smithi183 ceph-mon[33745]: osdmap e653: 8 total, 8 up, 8 in 2024-03-26T04:31:57.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:57 smithi016 ceph-mon[29101]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:31:57.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:57 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-130", "mode": "writeback"}]': finished 2024-03-26T04:31:57.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:57 smithi016 ceph-mon[29101]: osdmap e653: 8 total, 8 up, 8 in 2024-03-26T04:31:57.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:57 smithi016 ceph-mon[37215]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:31:57.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:57 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-130", "mode": "writeback"}]': finished 2024-03-26T04:31:57.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:57 smithi016 ceph-mon[37215]: osdmap e653: 8 total, 8 up, 8 in 2024-03-26T04:31:58.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:58 smithi016 ceph-mon[37215]: pgmap v1008: 268 pgs: 13 unknown, 255 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:31:58.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:31:58.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:58 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:31:58.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:58 smithi016 ceph-mon[29101]: pgmap v1008: 268 pgs: 13 unknown, 255 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:31:58.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:31:58.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:58 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:31:58.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:58 smithi183 ceph-mon[33745]: pgmap v1008: 268 pgs: 13 unknown, 255 active+clean; 582 KiB data, 637 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:31:58.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:31:58.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:58 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:31:59.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:31:58 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:31:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:31:59.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:59 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:31:59.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:59 smithi016 ceph-mon[29101]: osdmap e654: 8 total, 8 up, 8 in 2024-03-26T04:31:59.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:59 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-130"}]: dispatch 2024-03-26T04:31:59.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:31:59 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-130"}]: dispatch 2024-03-26T04:31:59.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:59 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:31:59.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:59 smithi016 ceph-mon[37215]: osdmap e654: 8 total, 8 up, 8 in 2024-03-26T04:31:59.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:59 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-130"}]: dispatch 2024-03-26T04:31:59.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:31:59 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-130"}]: dispatch 2024-03-26T04:31:59.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:59 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:31:59.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:59 smithi183 ceph-mon[33745]: osdmap e654: 8 total, 8 up, 8 in 2024-03-26T04:31:59.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:59 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-130"}]: dispatch 2024-03-26T04:31:59.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:31:59 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-130"}]: dispatch 2024-03-26T04:32:00.576 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:00 smithi016 ceph-mon[37215]: pgmap v1011: 268 pgs: 268 active+clean; 582 KiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:32:00.576 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:00 smithi016 ceph-mon[37215]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:32:00.577 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:00 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-130"}]': finished 2024-03-26T04:32:00.577 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:00 smithi016 ceph-mon[37215]: osdmap e655: 8 total, 8 up, 8 in 2024-03-26T04:32:00.577 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:00 smithi016 ceph-mon[29101]: pgmap v1011: 268 pgs: 268 active+clean; 582 KiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:32:00.577 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:00 smithi016 ceph-mon[29101]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:32:00.577 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:00 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-130"}]': finished 2024-03-26T04:32:00.577 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:00 smithi016 ceph-mon[29101]: osdmap e655: 8 total, 8 up, 8 in 2024-03-26T04:32:00.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:00 smithi183 ceph-mon[33745]: pgmap v1011: 268 pgs: 268 active+clean; 582 KiB data, 655 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:32:00.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:00 smithi183 ceph-mon[33745]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:32:00.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:00 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-130"}]': finished 2024-03-26T04:32:00.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:00 smithi183 ceph-mon[33745]: osdmap e655: 8 total, 8 up, 8 in 2024-03-26T04:32:01.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:01 smithi016 ceph-mon[37215]: osdmap e656: 8 total, 8 up, 8 in 2024-03-26T04:32:01.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:01 smithi016 ceph-mon[29101]: osdmap e656: 8 total, 8 up, 8 in 2024-03-26T04:32:01.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:01 smithi183 ceph-mon[33745]: osdmap e656: 8 total, 8 up, 8 in 2024-03-26T04:32:02.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:02 smithi016 ceph-mon[29101]: pgmap v1014: 236 pgs: 236 active+clean; 582 KiB data, 655 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:32:02.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:02 smithi016 ceph-mon[29101]: osdmap e657: 8 total, 8 up, 8 in 2024-03-26T04:32:02.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:02 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:32:02.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:02 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:32:02.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:02 smithi016 ceph-mon[37215]: pgmap v1014: 236 pgs: 236 active+clean; 582 KiB data, 655 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:32:02.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:02 smithi016 ceph-mon[37215]: osdmap e657: 8 total, 8 up, 8 in 2024-03-26T04:32:02.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:02 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:32:02.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:02 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:32:02.678 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:02 smithi183 ceph-mon[33745]: pgmap v1014: 236 pgs: 236 active+clean; 582 KiB data, 655 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:32:02.678 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:02 smithi183 ceph-mon[33745]: osdmap e657: 8 total, 8 up, 8 in 2024-03-26T04:32:02.679 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:02 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:32:02.679 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:02 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-132","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:32:03.041 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:32:02 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:32:03.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:03 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:32:03.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:03 smithi016 ceph-mon[37215]: osdmap e658: 8 total, 8 up, 8 in 2024-03-26T04:32:03.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:03 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:32:03.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:03 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:32:03.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:03 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:32:03.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:03 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:32:03.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:03 smithi016 ceph-mon[29101]: osdmap e658: 8 total, 8 up, 8 in 2024-03-26T04:32:03.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:03 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:32:03.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:03 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:32:03.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:03 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:32:03.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:03 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-132","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:32:03.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:03 smithi183 ceph-mon[33745]: osdmap e658: 8 total, 8 up, 8 in 2024-03-26T04:32:03.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:03 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:32:03.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:03 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-132", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:32:03.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:03 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:32:04.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:04 smithi016 ceph-mon[37215]: pgmap v1017: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:32:04.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:04 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-132", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:32:04.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:04 smithi016 ceph-mon[37215]: osdmap e659: 8 total, 8 up, 8 in 2024-03-26T04:32:04.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:04 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-132"}]: dispatch 2024-03-26T04:32:04.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:04 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-132"}]: dispatch 2024-03-26T04:32:04.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:04 smithi016 ceph-mon[29101]: pgmap v1017: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:32:04.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:04 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-132", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:32:04.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:04 smithi016 ceph-mon[29101]: osdmap e659: 8 total, 8 up, 8 in 2024-03-26T04:32:04.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:04 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-132"}]: dispatch 2024-03-26T04:32:04.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:04 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-132"}]: dispatch 2024-03-26T04:32:04.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:04 smithi183 ceph-mon[33745]: pgmap v1017: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:32:04.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:04 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-132", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:32:04.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:04 smithi183 ceph-mon[33745]: osdmap e659: 8 total, 8 up, 8 in 2024-03-26T04:32:04.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:04 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-132"}]: dispatch 2024-03-26T04:32:04.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:04 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-132"}]: dispatch 2024-03-26T04:32:05.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:05 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-132"}]': finished 2024-03-26T04:32:05.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:05 smithi016 ceph-mon[37215]: osdmap e660: 8 total, 8 up, 8 in 2024-03-26T04:32:05.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:05 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-132", "mode": "writeback"}]: dispatch 2024-03-26T04:32:05.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:05 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-132", "mode": "writeback"}]: dispatch 2024-03-26T04:32:05.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:05 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-132"}]': finished 2024-03-26T04:32:05.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:05 smithi016 ceph-mon[29101]: osdmap e660: 8 total, 8 up, 8 in 2024-03-26T04:32:05.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:05 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-132", "mode": "writeback"}]: dispatch 2024-03-26T04:32:05.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:05 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-132", "mode": "writeback"}]: dispatch 2024-03-26T04:32:05.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:05 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-132"}]': finished 2024-03-26T04:32:05.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:05 smithi183 ceph-mon[33745]: osdmap e660: 8 total, 8 up, 8 in 2024-03-26T04:32:05.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:05 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-132", "mode": "writeback"}]: dispatch 2024-03-26T04:32:05.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:05 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-132", "mode": "writeback"}]: dispatch 2024-03-26T04:32:06.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:06 smithi016 ceph-mon[29101]: pgmap v1020: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:32:06.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:06 smithi016 ceph-mon[29101]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:32:06.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:06 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-132", "mode": "writeback"}]': finished 2024-03-26T04:32:06.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:06 smithi016 ceph-mon[29101]: osdmap e661: 8 total, 8 up, 8 in 2024-03-26T04:32:06.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:06 smithi016 ceph-mon[37215]: pgmap v1020: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:32:06.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:06 smithi016 ceph-mon[37215]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:32:06.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:06 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-132", "mode": "writeback"}]': finished 2024-03-26T04:32:06.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:06 smithi016 ceph-mon[37215]: osdmap e661: 8 total, 8 up, 8 in 2024-03-26T04:32:06.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:06 smithi183 ceph-mon[33745]: pgmap v1020: 268 pgs: 32 creating+peering, 236 active+clean; 582 KiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:32:06.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:06 smithi183 ceph-mon[33745]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:32:06.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:06 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-132", "mode": "writeback"}]': finished 2024-03-26T04:32:06.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:06 smithi183 ceph-mon[33745]: osdmap e661: 8 total, 8 up, 8 in 2024-03-26T04:32:07.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:07 smithi016 ceph-mon[37215]: osdmap e662: 8 total, 8 up, 8 in 2024-03-26T04:32:07.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:07 smithi016 ceph-mon[29101]: osdmap e662: 8 total, 8 up, 8 in 2024-03-26T04:32:07.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:07 smithi183 ceph-mon[33745]: osdmap e662: 8 total, 8 up, 8 in 2024-03-26T04:32:08.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:08 smithi016 ceph-mon[29101]: pgmap v1023: 268 pgs: 9 creating+peering, 259 active+clean; 582 KiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:32:08.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:08 smithi016 ceph-mon[29101]: osdmap e663: 8 total, 8 up, 8 in 2024-03-26T04:32:08.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:08 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:32:08.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:08 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:32:08.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:08 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:32:08.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:08 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:32:08.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:08 smithi016 ceph-mon[37215]: pgmap v1023: 268 pgs: 9 creating+peering, 259 active+clean; 582 KiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:32:08.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:08 smithi016 ceph-mon[37215]: osdmap e663: 8 total, 8 up, 8 in 2024-03-26T04:32:08.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:08 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:32:08.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:08 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:32:08.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:08 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:32:08.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:08 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:32:08.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:08 smithi183 ceph-mon[33745]: pgmap v1023: 268 pgs: 9 creating+peering, 259 active+clean; 582 KiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:32:08.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:08 smithi183 ceph-mon[33745]: osdmap e663: 8 total, 8 up, 8 in 2024-03-26T04:32:08.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:08 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:32:08.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:08 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:32:08.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:08 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:32:08.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:08 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:32:09.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:32:08 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:32:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:32:09.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:09 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:32:09.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:09 smithi016 ceph-mon[37215]: osdmap e664: 8 total, 8 up, 8 in 2024-03-26T04:32:09.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:09 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-132"}]: dispatch 2024-03-26T04:32:09.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:09 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-132"}]: dispatch 2024-03-26T04:32:09.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:09 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:32:09.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:09 smithi016 ceph-mon[29101]: osdmap e664: 8 total, 8 up, 8 in 2024-03-26T04:32:09.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:09 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-132"}]: dispatch 2024-03-26T04:32:09.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:09 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-132"}]: dispatch 2024-03-26T04:32:09.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:09 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:32:09.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:09 smithi183 ceph-mon[33745]: osdmap e664: 8 total, 8 up, 8 in 2024-03-26T04:32:09.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:09 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-132"}]: dispatch 2024-03-26T04:32:09.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:09 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-132"}]: dispatch 2024-03-26T04:32:10.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:10 smithi016 ceph-mon[29101]: pgmap v1026: 268 pgs: 268 active+clean; 582 KiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:32:10.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:10 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-132"}]': finished 2024-03-26T04:32:10.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:10 smithi016 ceph-mon[29101]: osdmap e665: 8 total, 8 up, 8 in 2024-03-26T04:32:10.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:10 smithi016 ceph-mon[37215]: pgmap v1026: 268 pgs: 268 active+clean; 582 KiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:32:10.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:10 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-132"}]': finished 2024-03-26T04:32:10.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:10 smithi016 ceph-mon[37215]: osdmap e665: 8 total, 8 up, 8 in 2024-03-26T04:32:10.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:10 smithi183 ceph-mon[33745]: pgmap v1026: 268 pgs: 268 active+clean; 582 KiB data, 656 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:32:10.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:10 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-132"}]': finished 2024-03-26T04:32:10.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:10 smithi183 ceph-mon[33745]: osdmap e665: 8 total, 8 up, 8 in 2024-03-26T04:32:11.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:11 smithi016 ceph-mon[29101]: osdmap e666: 8 total, 8 up, 8 in 2024-03-26T04:32:11.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:11 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:32:11.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:11 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:32:11.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:11 smithi016 ceph-mon[37215]: osdmap e666: 8 total, 8 up, 8 in 2024-03-26T04:32:11.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:11 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:32:11.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:11 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:32:11.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:11 smithi183 ceph-mon[33745]: osdmap e666: 8 total, 8 up, 8 in 2024-03-26T04:32:11.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:11 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:32:11.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:11 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:32:12.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:12 smithi016 ceph-mon[29101]: pgmap v1029: 268 pgs: 268 active+clean; 582 KiB data, 656 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:32:12.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:12 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:32:12.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:12 smithi016 ceph-mon[29101]: osdmap e667: 8 total, 8 up, 8 in 2024-03-26T04:32:12.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:12 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-132"}]: dispatch 2024-03-26T04:32:12.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:12 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-132"}]: dispatch 2024-03-26T04:32:12.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:12 smithi016 ceph-mon[37215]: pgmap v1029: 268 pgs: 268 active+clean; 582 KiB data, 656 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:32:12.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:12 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:32:12.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:12 smithi016 ceph-mon[37215]: osdmap e667: 8 total, 8 up, 8 in 2024-03-26T04:32:12.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:12 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-132"}]: dispatch 2024-03-26T04:32:12.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:12 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-132"}]: dispatch 2024-03-26T04:32:12.689 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:12 smithi183 ceph-mon[33745]: pgmap v1029: 268 pgs: 268 active+clean; 582 KiB data, 656 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:32:12.689 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:12 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:32:12.690 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:12 smithi183 ceph-mon[33745]: osdmap e667: 8 total, 8 up, 8 in 2024-03-26T04:32:12.690 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:12 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-132"}]: dispatch 2024-03-26T04:32:12.690 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:12 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-132"}]: dispatch 2024-03-26T04:32:13.041 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:32:12 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:32:13.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:13 smithi016 ceph-mon[29101]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:32:13.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:13 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-132"}]': finished 2024-03-26T04:32:13.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:13 smithi016 ceph-mon[29101]: osdmap e668: 8 total, 8 up, 8 in 2024-03-26T04:32:13.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:13 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:32:13.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:13 smithi016 ceph-mon[37215]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:32:13.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:13 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-132"}]': finished 2024-03-26T04:32:13.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:13 smithi016 ceph-mon[37215]: osdmap e668: 8 total, 8 up, 8 in 2024-03-26T04:32:13.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:13 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:32:13.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:13 smithi183 ceph-mon[33745]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:32:13.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:13 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-132"}]': finished 2024-03-26T04:32:13.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:13 smithi183 ceph-mon[33745]: osdmap e668: 8 total, 8 up, 8 in 2024-03-26T04:32:13.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:13 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:32:14.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:14 smithi016 ceph-mon[37215]: pgmap v1032: 268 pgs: 268 active+clean; 582 KiB data, 657 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-26T04:32:14.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:14 smithi016 ceph-mon[37215]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:32:14.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:14 smithi016 ceph-mon[37215]: osdmap e669: 8 total, 8 up, 8 in 2024-03-26T04:32:14.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:14 smithi016 ceph-mon[29101]: pgmap v1032: 268 pgs: 268 active+clean; 582 KiB data, 657 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-26T04:32:14.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:14 smithi016 ceph-mon[29101]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:32:14.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:14 smithi016 ceph-mon[29101]: osdmap e669: 8 total, 8 up, 8 in 2024-03-26T04:32:14.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:14 smithi183 ceph-mon[33745]: pgmap v1032: 268 pgs: 268 active+clean; 582 KiB data, 657 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 1023 B/s wr, 3 op/s 2024-03-26T04:32:14.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:14 smithi183 ceph-mon[33745]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:32:14.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:14 smithi183 ceph-mon[33745]: osdmap e669: 8 total, 8 up, 8 in 2024-03-26T04:32:15.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:15 smithi183 ceph-mon[33745]: osdmap e670: 8 total, 8 up, 8 in 2024-03-26T04:32:15.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:15 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:32:15.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:15 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:32:15.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:15 smithi016 ceph-mon[37215]: osdmap e670: 8 total, 8 up, 8 in 2024-03-26T04:32:15.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:15 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:32:15.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:15 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:32:15.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:15 smithi016 ceph-mon[29101]: osdmap e670: 8 total, 8 up, 8 in 2024-03-26T04:32:15.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:15 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:32:15.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:15 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-134","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:32:16.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:16 smithi183 ceph-mon[33745]: pgmap v1035: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 657 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:32:16.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:16 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:32:16.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:16 smithi183 ceph-mon[33745]: osdmap e671: 8 total, 8 up, 8 in 2024-03-26T04:32:16.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:16 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:32:16.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:16 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:32:16.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:16 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-134", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:32:16.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:16 smithi183 ceph-mon[33745]: osdmap e672: 8 total, 8 up, 8 in 2024-03-26T04:32:16.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:16 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-134"}]: dispatch 2024-03-26T04:32:16.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:16 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-134"}]: dispatch 2024-03-26T04:32:16.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:16 smithi016 ceph-mon[29101]: pgmap v1035: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 657 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:32:16.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:16 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:32:16.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:16 smithi016 ceph-mon[29101]: osdmap e671: 8 total, 8 up, 8 in 2024-03-26T04:32:16.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:16 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:32:16.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:16 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:32:16.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:16 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-134", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:32:16.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:16 smithi016 ceph-mon[29101]: osdmap e672: 8 total, 8 up, 8 in 2024-03-26T04:32:16.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:16 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-134"}]: dispatch 2024-03-26T04:32:16.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:16 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-134"}]: dispatch 2024-03-26T04:32:16.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:16 smithi016 ceph-mon[37215]: pgmap v1035: 268 pgs: 32 unknown, 236 active+clean; 582 KiB data, 657 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:32:16.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:16 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-134","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:32:16.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:16 smithi016 ceph-mon[37215]: osdmap e671: 8 total, 8 up, 8 in 2024-03-26T04:32:16.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:16 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:32:16.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:16 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-134", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:32:16.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:16 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-134", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:32:16.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:16 smithi016 ceph-mon[37215]: osdmap e672: 8 total, 8 up, 8 in 2024-03-26T04:32:16.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:16 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-134"}]: dispatch 2024-03-26T04:32:16.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:16 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-134"}]: dispatch 2024-03-26T04:32:18.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:18 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-134"}]': finished 2024-03-26T04:32:18.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:18 smithi183 ceph-mon[33745]: osdmap e673: 8 total, 8 up, 8 in 2024-03-26T04:32:18.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:18 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-134", "mode": "writeback"}]: dispatch 2024-03-26T04:32:18.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:18 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-134", "mode": "writeback"}]: dispatch 2024-03-26T04:32:18.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:18 smithi183 ceph-mon[33745]: pgmap v1039: 268 pgs: 9 unknown, 259 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T04:32:18.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:18 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-134"}]': finished 2024-03-26T04:32:18.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:18 smithi016 ceph-mon[37215]: osdmap e673: 8 total, 8 up, 8 in 2024-03-26T04:32:18.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:18 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-134", "mode": "writeback"}]: dispatch 2024-03-26T04:32:18.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:18 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-134", "mode": "writeback"}]: dispatch 2024-03-26T04:32:18.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:18 smithi016 ceph-mon[37215]: pgmap v1039: 268 pgs: 9 unknown, 259 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T04:32:18.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:18 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-134"}]': finished 2024-03-26T04:32:18.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:18 smithi016 ceph-mon[29101]: osdmap e673: 8 total, 8 up, 8 in 2024-03-26T04:32:18.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:18 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-134", "mode": "writeback"}]: dispatch 2024-03-26T04:32:18.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:18 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-134", "mode": "writeback"}]: dispatch 2024-03-26T04:32:18.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:18 smithi016 ceph-mon[29101]: pgmap v1039: 268 pgs: 9 unknown, 259 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T04:32:19.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:32:18 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:32:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:32:19.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:19 smithi183 ceph-mon[33745]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:32:19.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:19 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-134", "mode": "writeback"}]': finished 2024-03-26T04:32:19.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:19 smithi183 ceph-mon[33745]: osdmap e674: 8 total, 8 up, 8 in 2024-03-26T04:32:19.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:19 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:32:19.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:19 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:32:19.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:19 smithi016 ceph-mon[37215]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:32:19.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:19 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-134", "mode": "writeback"}]': finished 2024-03-26T04:32:19.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:19 smithi016 ceph-mon[37215]: osdmap e674: 8 total, 8 up, 8 in 2024-03-26T04:32:19.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:19 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:32:19.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:19 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:32:19.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:19 smithi016 ceph-mon[29101]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:32:19.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:19 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-134", "mode": "writeback"}]': finished 2024-03-26T04:32:19.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:19 smithi016 ceph-mon[29101]: osdmap e674: 8 total, 8 up, 8 in 2024-03-26T04:32:19.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:19 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:32:19.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:19 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:32:20.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:20 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:32:20.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:20 smithi183 ceph-mon[33745]: osdmap e675: 8 total, 8 up, 8 in 2024-03-26T04:32:20.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:20 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-134"}]: dispatch 2024-03-26T04:32:20.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:20 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-134"}]: dispatch 2024-03-26T04:32:20.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:20 smithi183 ceph-mon[33745]: pgmap v1042: 268 pgs: 268 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T04:32:20.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:20 smithi183 ceph-mon[33745]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:32:20.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:20 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-134"}]': finished 2024-03-26T04:32:20.543 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:20 smithi183 ceph-mon[33745]: osdmap e676: 8 total, 8 up, 8 in 2024-03-26T04:32:20.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:20 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:32:20.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:20 smithi016 ceph-mon[29101]: osdmap e675: 8 total, 8 up, 8 in 2024-03-26T04:32:20.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:20 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-134"}]: dispatch 2024-03-26T04:32:20.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:20 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-134"}]: dispatch 2024-03-26T04:32:20.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:20 smithi016 ceph-mon[29101]: pgmap v1042: 268 pgs: 268 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T04:32:20.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:20 smithi016 ceph-mon[29101]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:32:20.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:20 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-134"}]': finished 2024-03-26T04:32:20.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:20 smithi016 ceph-mon[29101]: osdmap e676: 8 total, 8 up, 8 in 2024-03-26T04:32:20.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:20 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:32:20.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:20 smithi016 ceph-mon[37215]: osdmap e675: 8 total, 8 up, 8 in 2024-03-26T04:32:20.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:20 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-134"}]: dispatch 2024-03-26T04:32:20.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:20 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-134"}]: dispatch 2024-03-26T04:32:20.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:20 smithi016 ceph-mon[37215]: pgmap v1042: 268 pgs: 268 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T04:32:20.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:20 smithi016 ceph-mon[37215]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:32:20.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:20 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-134"}]': finished 2024-03-26T04:32:20.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:20 smithi016 ceph-mon[37215]: osdmap e676: 8 total, 8 up, 8 in 2024-03-26T04:32:21.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:21 smithi183 ceph-mon[33745]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:32:21.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:21 smithi016 ceph-mon[37215]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:32:21.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:21 smithi016 ceph-mon[29101]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:32:22.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:22 smithi183 ceph-mon[33745]: osdmap e677: 8 total, 8 up, 8 in 2024-03-26T04:32:22.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:22 smithi183 ceph-mon[33745]: pgmap v1045: 236 pgs: 236 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:32:22.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:22 smithi016 ceph-mon[29101]: osdmap e677: 8 total, 8 up, 8 in 2024-03-26T04:32:22.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:22 smithi016 ceph-mon[29101]: pgmap v1045: 236 pgs: 236 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:32:22.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:22 smithi016 ceph-mon[37215]: osdmap e677: 8 total, 8 up, 8 in 2024-03-26T04:32:22.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:22 smithi016 ceph-mon[37215]: pgmap v1045: 236 pgs: 236 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:32:23.041 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:32:22 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:32:23.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:23 smithi183 ceph-mon[33745]: osdmap e678: 8 total, 8 up, 8 in 2024-03-26T04:32:23.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:23 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:32:23.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:23 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:32:23.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:23 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:32:23.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:23 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:32:23.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:23 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:32:23.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:23 smithi016 ceph-mon[37215]: osdmap e678: 8 total, 8 up, 8 in 2024-03-26T04:32:23.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:23 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:32:23.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:23 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:32:23.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:23 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:32:23.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:23 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:32:23.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:23 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:32:23.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:23 smithi016 ceph-mon[29101]: osdmap e678: 8 total, 8 up, 8 in 2024-03-26T04:32:23.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:23 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:32:23.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:23 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-136","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:32:23.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:23 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:32:23.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:23 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:32:23.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:23 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:32:24.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:24 smithi183 ceph-mon[33745]: pgmap v1047: 268 pgs: 6 creating+peering, 26 unknown, 236 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T04:32:24.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:24 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:32:24.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:24 smithi183 ceph-mon[33745]: osdmap e679: 8 total, 8 up, 8 in 2024-03-26T04:32:24.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:24 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:32:24.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:24 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:32:24.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:24 smithi016 ceph-mon[29101]: pgmap v1047: 268 pgs: 6 creating+peering, 26 unknown, 236 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T04:32:24.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:24 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:32:24.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:24 smithi016 ceph-mon[29101]: osdmap e679: 8 total, 8 up, 8 in 2024-03-26T04:32:24.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:24 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:32:24.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:24 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:32:24.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:24 smithi016 ceph-mon[37215]: pgmap v1047: 268 pgs: 6 creating+peering, 26 unknown, 236 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T04:32:24.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:24 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-136","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:32:24.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:24 smithi016 ceph-mon[37215]: osdmap e679: 8 total, 8 up, 8 in 2024-03-26T04:32:24.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:24 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:32:24.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:24 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-136", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:32:25.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:25 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-136", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:32:25.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:25 smithi183 ceph-mon[33745]: osdmap e680: 8 total, 8 up, 8 in 2024-03-26T04:32:25.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:25 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-136"}]: dispatch 2024-03-26T04:32:25.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:25 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-136"}]: dispatch 2024-03-26T04:32:25.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:25 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-136"}]': finished 2024-03-26T04:32:25.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:25 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-136", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:32:25.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:25 smithi016 ceph-mon[29101]: osdmap e680: 8 total, 8 up, 8 in 2024-03-26T04:32:25.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:25 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-136"}]: dispatch 2024-03-26T04:32:25.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:25 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-136"}]: dispatch 2024-03-26T04:32:25.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:25 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-136"}]': finished 2024-03-26T04:32:25.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:25 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-136", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:32:25.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:25 smithi016 ceph-mon[37215]: osdmap e680: 8 total, 8 up, 8 in 2024-03-26T04:32:25.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:25 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-136"}]: dispatch 2024-03-26T04:32:25.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:25 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-136"}]: dispatch 2024-03-26T04:32:25.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:25 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-136"}]': finished 2024-03-26T04:32:26.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:26 smithi183 ceph-mon[33745]: pgmap v1050: 268 pgs: 6 creating+peering, 26 unknown, 236 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T04:32:26.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:26 smithi183 ceph-mon[33745]: osdmap e681: 8 total, 8 up, 8 in 2024-03-26T04:32:26.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:26 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-136", "mode": "writeback"}]: dispatch 2024-03-26T04:32:26.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:26 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-136", "mode": "writeback"}]: dispatch 2024-03-26T04:32:26.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:26 smithi183 ceph-mon[33745]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:32:26.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:26 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-136", "mode": "writeback"}]': finished 2024-03-26T04:32:26.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:26 smithi183 ceph-mon[33745]: osdmap e682: 8 total, 8 up, 8 in 2024-03-26T04:32:26.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:26 smithi016 ceph-mon[29101]: pgmap v1050: 268 pgs: 6 creating+peering, 26 unknown, 236 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T04:32:26.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:26 smithi016 ceph-mon[29101]: osdmap e681: 8 total, 8 up, 8 in 2024-03-26T04:32:26.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:26 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-136", "mode": "writeback"}]: dispatch 2024-03-26T04:32:26.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:26 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-136", "mode": "writeback"}]: dispatch 2024-03-26T04:32:26.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:26 smithi016 ceph-mon[29101]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:32:26.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:26 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-136", "mode": "writeback"}]': finished 2024-03-26T04:32:26.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:26 smithi016 ceph-mon[29101]: osdmap e682: 8 total, 8 up, 8 in 2024-03-26T04:32:26.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:26 smithi016 ceph-mon[37215]: pgmap v1050: 268 pgs: 6 creating+peering, 26 unknown, 236 active+clean; 582 KiB data, 658 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.2 KiB/s wr, 2 op/s 2024-03-26T04:32:26.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:26 smithi016 ceph-mon[37215]: osdmap e681: 8 total, 8 up, 8 in 2024-03-26T04:32:26.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:26 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-136", "mode": "writeback"}]: dispatch 2024-03-26T04:32:26.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:26 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-136", "mode": "writeback"}]: dispatch 2024-03-26T04:32:26.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:26 smithi016 ceph-mon[37215]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:32:26.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:26 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-136", "mode": "writeback"}]': finished 2024-03-26T04:32:26.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:26 smithi016 ceph-mon[37215]: osdmap e682: 8 total, 8 up, 8 in 2024-03-26T04:32:27.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:27 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:32:27.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:27 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:32:27.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:27 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:32:27.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:27 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:32:27.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:27 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:32:27.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:27 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:32:28.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:28 smithi183 ceph-mon[33745]: pgmap v1053: 268 pgs: 6 creating+peering, 7 unknown, 255 active+clean; 4.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s 2024-03-26T04:32:28.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:28 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:32:28.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:28 smithi183 ceph-mon[33745]: osdmap e683: 8 total, 8 up, 8 in 2024-03-26T04:32:28.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:28 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-136"}]: dispatch 2024-03-26T04:32:28.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:28 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-136"}]: dispatch 2024-03-26T04:32:28.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:28 smithi016 ceph-mon[37215]: pgmap v1053: 268 pgs: 6 creating+peering, 7 unknown, 255 active+clean; 4.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s 2024-03-26T04:32:28.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:28 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:32:28.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:28 smithi016 ceph-mon[37215]: osdmap e683: 8 total, 8 up, 8 in 2024-03-26T04:32:28.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:28 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-136"}]: dispatch 2024-03-26T04:32:28.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:28 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-136"}]: dispatch 2024-03-26T04:32:28.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:28 smithi016 ceph-mon[29101]: pgmap v1053: 268 pgs: 6 creating+peering, 7 unknown, 255 active+clean; 4.4 MiB data, 670 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s 2024-03-26T04:32:28.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:28 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:32:28.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:28 smithi016 ceph-mon[29101]: osdmap e683: 8 total, 8 up, 8 in 2024-03-26T04:32:28.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:28 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-136"}]: dispatch 2024-03-26T04:32:28.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:28 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-136"}]: dispatch 2024-03-26T04:32:29.175 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:32:28 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:32:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:32:29.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:29 smithi183 ceph-mon[33745]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:32:29.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:29 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-136"}]': finished 2024-03-26T04:32:29.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:29 smithi183 ceph-mon[33745]: osdmap e684: 8 total, 8 up, 8 in 2024-03-26T04:32:29.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:29 smithi016 ceph-mon[29101]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:32:29.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:29 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-136"}]': finished 2024-03-26T04:32:29.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:29 smithi016 ceph-mon[29101]: osdmap e684: 8 total, 8 up, 8 in 2024-03-26T04:32:29.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:29 smithi016 ceph-mon[37215]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:32:29.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:29 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-136"}]': finished 2024-03-26T04:32:29.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:29 smithi016 ceph-mon[37215]: osdmap e684: 8 total, 8 up, 8 in 2024-03-26T04:32:30.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:30 smithi183 ceph-mon[33745]: pgmap v1056: 268 pgs: 268 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s 2024-03-26T04:32:30.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:30 smithi183 ceph-mon[33745]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:32:30.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:30 smithi183 ceph-mon[33745]: osdmap e685: 8 total, 8 up, 8 in 2024-03-26T04:32:30.564 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:30 smithi016 ceph-mon[29101]: pgmap v1056: 268 pgs: 268 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s 2024-03-26T04:32:30.564 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:30 smithi016 ceph-mon[29101]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:32:30.564 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:30 smithi016 ceph-mon[29101]: osdmap e685: 8 total, 8 up, 8 in 2024-03-26T04:32:30.564 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:30 smithi016 ceph-mon[37215]: pgmap v1056: 268 pgs: 268 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.9 MiB/s wr, 1 op/s 2024-03-26T04:32:30.564 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:30 smithi016 ceph-mon[37215]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:32:30.564 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:30 smithi016 ceph-mon[37215]: osdmap e685: 8 total, 8 up, 8 in 2024-03-26T04:32:31.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:31 smithi016 ceph-mon[29101]: osdmap e686: 8 total, 8 up, 8 in 2024-03-26T04:32:31.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:31 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:32:31.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:31 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:32:31.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:31 smithi016 ceph-mon[37215]: osdmap e686: 8 total, 8 up, 8 in 2024-03-26T04:32:31.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:31 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:32:31.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:31 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:32:31.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:31 smithi183 ceph-mon[33745]: osdmap e686: 8 total, 8 up, 8 in 2024-03-26T04:32:31.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:31 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:32:31.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:31 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-138","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:32:32.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:32 smithi016 ceph-mon[37215]: pgmap v1059: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:32:32.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:32 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:32:32.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:32 smithi016 ceph-mon[37215]: osdmap e687: 8 total, 8 up, 8 in 2024-03-26T04:32:32.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:32 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:32:32.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:32 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:32:32.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:32 smithi016 ceph-mon[29101]: pgmap v1059: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:32:32.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:32 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:32:32.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:32 smithi016 ceph-mon[29101]: osdmap e687: 8 total, 8 up, 8 in 2024-03-26T04:32:32.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:32 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:32:32.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:32 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:32:32.710 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:32 smithi183 ceph-mon[33745]: pgmap v1059: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 677 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:32:32.710 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:32 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-138","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:32:32.711 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:32 smithi183 ceph-mon[33745]: osdmap e687: 8 total, 8 up, 8 in 2024-03-26T04:32:32.711 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:32 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:32:32.711 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:32 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-138", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:32:33.041 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:32:32 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:32:33.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:33 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-138", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:32:33.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:33 smithi016 ceph-mon[37215]: osdmap e688: 8 total, 8 up, 8 in 2024-03-26T04:32:33.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:33 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:32:33.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:33 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:32:33.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:33 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:32:33.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:33 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-138","var": "hit_set_count","val": "2"}]': finished 2024-03-26T04:32:33.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:33 smithi016 ceph-mon[37215]: osdmap e689: 8 total, 8 up, 8 in 2024-03-26T04:32:33.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:33 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-138", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:32:33.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:33 smithi016 ceph-mon[29101]: osdmap e688: 8 total, 8 up, 8 in 2024-03-26T04:32:33.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:33 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:32:33.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:33 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:32:33.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:33 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:32:33.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:33 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-138","var": "hit_set_count","val": "2"}]': finished 2024-03-26T04:32:33.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:33 smithi016 ceph-mon[29101]: osdmap e689: 8 total, 8 up, 8 in 2024-03-26T04:32:33.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:33 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-138", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:32:33.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:33 smithi183 ceph-mon[33745]: osdmap e688: 8 total, 8 up, 8 in 2024-03-26T04:32:33.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:33 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:32:33.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:33 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-138","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:32:33.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:33 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:32:33.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:33 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-138","var": "hit_set_count","val": "2"}]': finished 2024-03-26T04:32:33.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:33 smithi183 ceph-mon[33745]: osdmap e689: 8 total, 8 up, 8 in 2024-03-26T04:32:34.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:34 smithi016 ceph-mon[37215]: pgmap v1062: 268 pgs: 268 active+clean; 4.4 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:32:34.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:34 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:32:34.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:34 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:32:34.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:34 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-138","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:32:34.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:34 smithi016 ceph-mon[37215]: osdmap e690: 8 total, 8 up, 8 in 2024-03-26T04:32:34.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:34 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T04:32:34.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:34 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T04:32:34.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:34 smithi016 ceph-mon[29101]: pgmap v1062: 268 pgs: 268 active+clean; 4.4 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:32:34.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:34 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:32:34.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:34 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:32:34.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:34 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-138","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:32:34.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:34 smithi016 ceph-mon[29101]: osdmap e690: 8 total, 8 up, 8 in 2024-03-26T04:32:34.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:34 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T04:32:34.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:34 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T04:32:34.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:34 smithi183 ceph-mon[33745]: pgmap v1062: 268 pgs: 268 active+clean; 4.4 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:32:34.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:34 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:32:34.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:34 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-138","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:32:34.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:34 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-138","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:32:34.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:34 smithi183 ceph-mon[33745]: osdmap e690: 8 total, 8 up, 8 in 2024-03-26T04:32:34.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:34 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T04:32:34.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:34 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-138","var": "hit_set_type","val": "explicit_object"}]: dispatch 2024-03-26T04:32:36.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:36 smithi016 ceph-mon[37215]: pgmap v1065: 268 pgs: 268 active+clean; 4.4 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:32:36.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:36 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T04:32:36.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:36 smithi016 ceph-mon[37215]: osdmap e691: 8 total, 8 up, 8 in 2024-03-26T04:32:36.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:36 smithi016 ceph-mon[37215]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:32:36.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:36 smithi016 ceph-mon[29101]: pgmap v1065: 268 pgs: 268 active+clean; 4.4 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:32:36.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:36 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T04:32:36.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:36 smithi016 ceph-mon[29101]: osdmap e691: 8 total, 8 up, 8 in 2024-03-26T04:32:36.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:36 smithi016 ceph-mon[29101]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:32:36.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:36 smithi183 ceph-mon[33745]: pgmap v1065: 268 pgs: 268 active+clean; 4.4 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:32:36.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:36 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-138","var": "hit_set_type","val": "explicit_object"}]': finished 2024-03-26T04:32:36.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:36 smithi183 ceph-mon[33745]: osdmap e691: 8 total, 8 up, 8 in 2024-03-26T04:32:36.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:36 smithi183 ceph-mon[33745]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:32:37.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:37 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:32:37.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:37 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:32:37.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:37 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-138"}]: dispatch 2024-03-26T04:32:37.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:37 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-138"}]: dispatch 2024-03-26T04:32:37.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:37 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:32:37.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:37 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:32:37.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:37 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-138"}]: dispatch 2024-03-26T04:32:37.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:37 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-138"}]: dispatch 2024-03-26T04:32:37.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:37 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:32:37.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:37 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:32:37.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:37 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-138"}]: dispatch 2024-03-26T04:32:37.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:37 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-138"}]: dispatch 2024-03-26T04:32:38.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:38 smithi016 ceph-mon[37215]: pgmap v1067: 268 pgs: 268 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T04:32:38.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:38 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-138"}]': finished 2024-03-26T04:32:38.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:38 smithi016 ceph-mon[37215]: osdmap e692: 8 total, 8 up, 8 in 2024-03-26T04:32:38.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:38 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:32:38.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:38 smithi016 ceph-mon[29101]: pgmap v1067: 268 pgs: 268 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T04:32:38.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:38 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-138"}]': finished 2024-03-26T04:32:38.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:38 smithi016 ceph-mon[29101]: osdmap e692: 8 total, 8 up, 8 in 2024-03-26T04:32:38.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:38 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:32:38.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:38 smithi183 ceph-mon[33745]: pgmap v1067: 268 pgs: 268 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T04:32:38.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:38 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-138"}]': finished 2024-03-26T04:32:38.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:38 smithi183 ceph-mon[33745]: osdmap e692: 8 total, 8 up, 8 in 2024-03-26T04:32:38.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:38 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:32:39.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:32:38 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:32:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:32:39.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:39 smithi016 ceph-mon[37215]: osdmap e693: 8 total, 8 up, 8 in 2024-03-26T04:32:39.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:39 smithi016 ceph-mon[29101]: osdmap e693: 8 total, 8 up, 8 in 2024-03-26T04:32:39.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:39 smithi183 ceph-mon[33745]: osdmap e693: 8 total, 8 up, 8 in 2024-03-26T04:32:40.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:40 smithi016 ceph-mon[29101]: pgmap v1070: 236 pgs: 236 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T04:32:40.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:40 smithi016 ceph-mon[29101]: osdmap e694: 8 total, 8 up, 8 in 2024-03-26T04:32:40.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:40 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:32:40.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:40 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:32:40.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:40 smithi016 ceph-mon[37215]: pgmap v1070: 236 pgs: 236 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T04:32:40.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:40 smithi016 ceph-mon[37215]: osdmap e694: 8 total, 8 up, 8 in 2024-03-26T04:32:40.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:40 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:32:40.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:40 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:32:40.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:40 smithi183 ceph-mon[33745]: pgmap v1070: 236 pgs: 236 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T04:32:40.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:40 smithi183 ceph-mon[33745]: osdmap e694: 8 total, 8 up, 8 in 2024-03-26T04:32:40.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:40 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:32:40.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:40 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-140","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:32:41.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:41 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:32:41.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:41 smithi016 ceph-mon[29101]: osdmap e695: 8 total, 8 up, 8 in 2024-03-26T04:32:41.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:41 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:32:41.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:41 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:32:41.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:41 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:32:41.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:41 smithi016 ceph-mon[37215]: osdmap e695: 8 total, 8 up, 8 in 2024-03-26T04:32:41.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:41 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:32:41.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:41 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:32:41.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:41 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-140","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:32:41.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:41 smithi183 ceph-mon[33745]: osdmap e695: 8 total, 8 up, 8 in 2024-03-26T04:32:41.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:41 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:32:41.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:41 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-140", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:32:42.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:42 smithi016 ceph-mon[29101]: pgmap v1073: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:32:42.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:42 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-140", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:32:42.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:42 smithi016 ceph-mon[29101]: osdmap e696: 8 total, 8 up, 8 in 2024-03-26T04:32:42.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:42 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T04:32:42.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:42 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T04:32:42.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:42 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:32:42.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:42 smithi016 ceph-mon[37215]: pgmap v1073: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:32:42.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:42 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-140", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:32:42.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:42 smithi016 ceph-mon[37215]: osdmap e696: 8 total, 8 up, 8 in 2024-03-26T04:32:42.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:42 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T04:32:42.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:42 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T04:32:42.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:42 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:32:42.721 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:42 smithi183 ceph-mon[33745]: pgmap v1073: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 678 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:32:42.721 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:42 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-140", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:32:42.721 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:42 smithi183 ceph-mon[33745]: osdmap e696: 8 total, 8 up, 8 in 2024-03-26T04:32:42.721 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:42 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T04:32:42.721 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:42 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-140","var": "hit_set_count","val": "3"}]: dispatch 2024-03-26T04:32:42.721 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:42 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:32:43.041 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:32:42 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:32:43.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:43 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-140","var": "hit_set_count","val": "3"}]': finished 2024-03-26T04:32:43.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:43 smithi016 ceph-mon[37215]: osdmap e697: 8 total, 8 up, 8 in 2024-03-26T04:32:43.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:43 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T04:32:43.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:43 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T04:32:43.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:43 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:32:43.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:43 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:32:43.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:43 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:32:43.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:43 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:32:43.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:43 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-140","var": "hit_set_period","val": "3"}]': finished 2024-03-26T04:32:43.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:43 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-140","var": "hit_set_count","val": "3"}]': finished 2024-03-26T04:32:43.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:43 smithi016 ceph-mon[29101]: osdmap e697: 8 total, 8 up, 8 in 2024-03-26T04:32:43.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:43 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T04:32:43.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:43 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T04:32:43.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:43 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:32:43.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:43 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:32:43.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:43 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:32:43.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:43 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:32:43.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:43 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-140","var": "hit_set_period","val": "3"}]': finished 2024-03-26T04:32:43.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:43 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-140","var": "hit_set_count","val": "3"}]': finished 2024-03-26T04:32:43.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:43 smithi183 ceph-mon[33745]: osdmap e697: 8 total, 8 up, 8 in 2024-03-26T04:32:43.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:43 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T04:32:43.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:43 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-140","var": "hit_set_period","val": "3"}]: dispatch 2024-03-26T04:32:43.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:43 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:32:43.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:43 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:32:43.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:43 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:32:43.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:43 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:32:43.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:43 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-140","var": "hit_set_period","val": "3"}]': finished 2024-03-26T04:32:44.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:44 smithi016 ceph-mon[29101]: pgmap v1076: 268 pgs: 268 active+clean; 4.4 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:32:44.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:44 smithi016 ceph-mon[29101]: osdmap e698: 8 total, 8 up, 8 in 2024-03-26T04:32:44.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:44 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:32:44.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:44 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:32:44.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:44 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-140","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T04:32:44.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:44 smithi016 ceph-mon[29101]: osdmap e699: 8 total, 8 up, 8 in 2024-03-26T04:32:44.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:44 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T04:32:44.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:44 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T04:32:44.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:44 smithi016 ceph-mon[37215]: pgmap v1076: 268 pgs: 268 active+clean; 4.4 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:32:44.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:44 smithi016 ceph-mon[37215]: osdmap e698: 8 total, 8 up, 8 in 2024-03-26T04:32:44.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:44 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:32:44.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:44 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:32:44.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:44 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-140","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T04:32:44.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:44 smithi016 ceph-mon[37215]: osdmap e699: 8 total, 8 up, 8 in 2024-03-26T04:32:44.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:44 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T04:32:44.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:44 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T04:32:44.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:44 smithi183 ceph-mon[33745]: pgmap v1076: 268 pgs: 268 active+clean; 4.4 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:32:44.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:44 smithi183 ceph-mon[33745]: osdmap e698: 8 total, 8 up, 8 in 2024-03-26T04:32:44.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:44 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:32:44.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:44 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-140","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:32:44.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:44 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-140","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T04:32:44.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:44 smithi183 ceph-mon[33745]: osdmap e699: 8 total, 8 up, 8 in 2024-03-26T04:32:44.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:44 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T04:32:44.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:44 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-140","var": "hit_set_fpp","val": ".01"}]: dispatch 2024-03-26T04:32:46.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:46 smithi016 ceph-mon[29101]: pgmap v1079: 268 pgs: 268 active+clean; 4.4 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:32:46.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:46 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-26T04:32:46.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:46 smithi016 ceph-mon[29101]: osdmap e700: 8 total, 8 up, 8 in 2024-03-26T04:32:46.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:46 smithi016 ceph-mon[37215]: pgmap v1079: 268 pgs: 268 active+clean; 4.4 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:32:46.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:46 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-26T04:32:46.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:46 smithi016 ceph-mon[37215]: osdmap e700: 8 total, 8 up, 8 in 2024-03-26T04:32:46.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:46 smithi183 ceph-mon[33745]: pgmap v1079: 268 pgs: 268 active+clean; 4.4 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:32:46.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:46 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-140","var": "hit_set_fpp","val": ".01"}]': finished 2024-03-26T04:32:46.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:46 smithi183 ceph-mon[33745]: osdmap e700: 8 total, 8 up, 8 in 2024-03-26T04:32:48.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:48 smithi016 ceph-mon[29101]: pgmap v1081: 268 pgs: 268 active+clean; 4.4 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T04:32:48.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:48 smithi016 ceph-mon[37215]: pgmap v1081: 268 pgs: 268 active+clean; 4.4 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T04:32:48.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:48 smithi183 ceph-mon[33745]: pgmap v1081: 268 pgs: 268 active+clean; 4.4 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T04:32:49.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:32:48 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:32:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:32:50.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:50 smithi016 ceph-mon[29101]: pgmap v1082: 268 pgs: 268 active+clean; 4.4 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 2.7 KiB/s wr, 1 op/s 2024-03-26T04:32:50.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:50 smithi016 ceph-mon[37215]: pgmap v1082: 268 pgs: 268 active+clean; 4.4 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 2.7 KiB/s wr, 1 op/s 2024-03-26T04:32:50.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:50 smithi183 ceph-mon[33745]: pgmap v1082: 268 pgs: 268 active+clean; 4.4 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 2.7 KiB/s wr, 1 op/s 2024-03-26T04:32:52.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:52 smithi016 ceph-mon[29101]: pgmap v1083: 268 pgs: 268 active+clean; 4.4 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 648 B/s rd, 2.0 KiB/s wr, 0 op/s 2024-03-26T04:32:52.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:52 smithi016 ceph-mon[37215]: pgmap v1083: 268 pgs: 268 active+clean; 4.4 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 648 B/s rd, 2.0 KiB/s wr, 0 op/s 2024-03-26T04:32:52.732 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:52 smithi183 ceph-mon[33745]: pgmap v1083: 268 pgs: 268 active+clean; 4.4 MiB data, 679 MiB used, 715 GiB / 715 GiB avail; 648 B/s rd, 2.0 KiB/s wr, 0 op/s 2024-03-26T04:32:53.041 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:32:52 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:32:53.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:53 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:32:53.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:53 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:32:53.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:53 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:32:54.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:54 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:32:54.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:54 smithi016 ceph-mon[29101]: pgmap v1084: 268 pgs: 268 active+clean; 4.4 MiB data, 683 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 6.5 KiB/s wr, 2 op/s 2024-03-26T04:32:54.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:54 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:32:54.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:54 smithi016 ceph-mon[37215]: pgmap v1084: 268 pgs: 268 active+clean; 4.4 MiB data, 683 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 6.5 KiB/s wr, 2 op/s 2024-03-26T04:32:54.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:54 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:32:54.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:54 smithi183 ceph-mon[33745]: pgmap v1084: 268 pgs: 268 active+clean; 4.4 MiB data, 683 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 6.5 KiB/s wr, 2 op/s 2024-03-26T04:32:56.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:56 smithi016 ceph-mon[29101]: pgmap v1085: 268 pgs: 268 active+clean; 4.4 MiB data, 683 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 5.8 KiB/s wr, 1 op/s 2024-03-26T04:32:56.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:56 smithi016 ceph-mon[37215]: pgmap v1085: 268 pgs: 268 active+clean; 4.4 MiB data, 683 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 5.8 KiB/s wr, 1 op/s 2024-03-26T04:32:56.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:56 smithi183 ceph-mon[33745]: pgmap v1085: 268 pgs: 268 active+clean; 4.4 MiB data, 683 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 5.8 KiB/s wr, 1 op/s 2024-03-26T04:32:58.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:58 smithi016 ceph-mon[37215]: pgmap v1086: 268 pgs: 268 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 4.9 KiB/s wr, 2 op/s 2024-03-26T04:32:58.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:32:58.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:58 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:32:58.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-140"}]: dispatch 2024-03-26T04:32:58.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:58 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-140"}]: dispatch 2024-03-26T04:32:58.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:58 smithi016 ceph-mon[29101]: pgmap v1086: 268 pgs: 268 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 4.9 KiB/s wr, 2 op/s 2024-03-26T04:32:58.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:32:58.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:58 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:32:58.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-140"}]: dispatch 2024-03-26T04:32:58.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:58 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-140"}]: dispatch 2024-03-26T04:32:58.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:58 smithi183 ceph-mon[33745]: pgmap v1086: 268 pgs: 268 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 4.9 KiB/s wr, 2 op/s 2024-03-26T04:32:58.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:32:58.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:58 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:32:58.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-140"}]: dispatch 2024-03-26T04:32:58.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:58 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-140"}]: dispatch 2024-03-26T04:32:59.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:32:58 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:32:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:32:59.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:59 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-140"}]': finished 2024-03-26T04:32:59.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:32:59 smithi016 ceph-mon[37215]: osdmap e701: 8 total, 8 up, 8 in 2024-03-26T04:32:59.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:59 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-140"}]': finished 2024-03-26T04:32:59.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:32:59 smithi016 ceph-mon[29101]: osdmap e701: 8 total, 8 up, 8 in 2024-03-26T04:32:59.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:59 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-140"}]': finished 2024-03-26T04:32:59.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:32:59 smithi183 ceph-mon[33745]: osdmap e701: 8 total, 8 up, 8 in 2024-03-26T04:33:00.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:00 smithi016 ceph-mon[37215]: pgmap v1088: 268 pgs: 268 active+clean; 4.5 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 8.3 KiB/s wr, 2 op/s 2024-03-26T04:33:00.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:00 smithi016 ceph-mon[37215]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:33:00.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:00 smithi016 ceph-mon[37215]: osdmap e702: 8 total, 8 up, 8 in 2024-03-26T04:33:00.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:00 smithi016 ceph-mon[29101]: pgmap v1088: 268 pgs: 268 active+clean; 4.5 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 8.3 KiB/s wr, 2 op/s 2024-03-26T04:33:00.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:00 smithi016 ceph-mon[29101]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:33:00.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:00 smithi016 ceph-mon[29101]: osdmap e702: 8 total, 8 up, 8 in 2024-03-26T04:33:00.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:00 smithi183 ceph-mon[33745]: pgmap v1088: 268 pgs: 268 active+clean; 4.5 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 8.3 KiB/s wr, 2 op/s 2024-03-26T04:33:00.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:00 smithi183 ceph-mon[33745]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:33:00.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:00 smithi183 ceph-mon[33745]: osdmap e702: 8 total, 8 up, 8 in 2024-03-26T04:33:01.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:01 smithi016 ceph-mon[37215]: osdmap e703: 8 total, 8 up, 8 in 2024-03-26T04:33:01.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:01 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:33:01.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:01 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:33:01.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:01 smithi016 ceph-mon[29101]: osdmap e703: 8 total, 8 up, 8 in 2024-03-26T04:33:01.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:01 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:33:01.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:01 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:33:01.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:01 smithi183 ceph-mon[33745]: osdmap e703: 8 total, 8 up, 8 in 2024-03-26T04:33:01.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:01 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:33:01.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:01 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-142","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:33:02.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:02 smithi016 ceph-mon[29101]: pgmap v1091: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:33:02.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:02 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:33:02.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:02 smithi016 ceph-mon[29101]: osdmap e704: 8 total, 8 up, 8 in 2024-03-26T04:33:02.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:02 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:33:02.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:02 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:33:02.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:02 smithi016 ceph-mon[37215]: pgmap v1091: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:33:02.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:02 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:33:02.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:02 smithi016 ceph-mon[37215]: osdmap e704: 8 total, 8 up, 8 in 2024-03-26T04:33:02.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:02 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:33:02.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:02 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:33:02.736 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:02 smithi183 ceph-mon[33745]: pgmap v1091: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:33:02.736 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:02 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-142","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:33:02.736 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:02 smithi183 ceph-mon[33745]: osdmap e704: 8 total, 8 up, 8 in 2024-03-26T04:33:02.737 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:02 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:33:02.737 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:02 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-142", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:33:03.041 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:33:02 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:33:03.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:03 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-142", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:33:03.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:03 smithi016 ceph-mon[37215]: osdmap e705: 8 total, 8 up, 8 in 2024-03-26T04:33:03.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:03 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-142"}]: dispatch 2024-03-26T04:33:03.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:03 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-142"}]: dispatch 2024-03-26T04:33:03.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:03 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-142", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:33:03.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:03 smithi016 ceph-mon[29101]: osdmap e705: 8 total, 8 up, 8 in 2024-03-26T04:33:03.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:03 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-142"}]: dispatch 2024-03-26T04:33:03.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:03 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-142"}]: dispatch 2024-03-26T04:33:03.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:03 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-142", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:33:03.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:03 smithi183 ceph-mon[33745]: osdmap e705: 8 total, 8 up, 8 in 2024-03-26T04:33:03.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:03 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-142"}]: dispatch 2024-03-26T04:33:03.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:03 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-142"}]: dispatch 2024-03-26T04:33:04.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:04 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:33:04.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:04 smithi016 ceph-mon[37215]: pgmap v1094: 268 pgs: 268 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T04:33:04.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:04 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-142"}]': finished 2024-03-26T04:33:04.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:04 smithi016 ceph-mon[37215]: osdmap e706: 8 total, 8 up, 8 in 2024-03-26T04:33:04.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:04 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-142", "mode": "writeback"}]: dispatch 2024-03-26T04:33:04.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:04 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-142", "mode": "writeback"}]: dispatch 2024-03-26T04:33:04.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:04 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:33:04.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:04 smithi016 ceph-mon[29101]: pgmap v1094: 268 pgs: 268 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T04:33:04.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:04 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-142"}]': finished 2024-03-26T04:33:04.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:04 smithi016 ceph-mon[29101]: osdmap e706: 8 total, 8 up, 8 in 2024-03-26T04:33:04.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:04 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-142", "mode": "writeback"}]: dispatch 2024-03-26T04:33:04.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:04 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-142", "mode": "writeback"}]: dispatch 2024-03-26T04:33:04.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:04 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:33:04.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:04 smithi183 ceph-mon[33745]: pgmap v1094: 268 pgs: 268 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-26T04:33:04.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:04 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-142"}]': finished 2024-03-26T04:33:04.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:04 smithi183 ceph-mon[33745]: osdmap e706: 8 total, 8 up, 8 in 2024-03-26T04:33:04.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:04 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-142", "mode": "writeback"}]: dispatch 2024-03-26T04:33:04.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:04 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-142", "mode": "writeback"}]: dispatch 2024-03-26T04:33:05.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:05 smithi183 ceph-mon[33745]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:33:05.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:05 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-142", "mode": "writeback"}]': finished 2024-03-26T04:33:05.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:05 smithi183 ceph-mon[33745]: osdmap e707: 8 total, 8 up, 8 in 2024-03-26T04:33:05.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:05 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:33:05.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:05 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:33:05.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:05 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "hit_set_count","val": "2"}]': finished 2024-03-26T04:33:05.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:05 smithi016 ceph-mon[37215]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:33:05.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:05 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-142", "mode": "writeback"}]': finished 2024-03-26T04:33:05.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:05 smithi016 ceph-mon[37215]: osdmap e707: 8 total, 8 up, 8 in 2024-03-26T04:33:05.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:05 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:33:05.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:05 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:33:05.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:05 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "hit_set_count","val": "2"}]': finished 2024-03-26T04:33:05.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:05 smithi016 ceph-mon[29101]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:33:05.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:05 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-142", "mode": "writeback"}]': finished 2024-03-26T04:33:05.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:05 smithi016 ceph-mon[29101]: osdmap e707: 8 total, 8 up, 8 in 2024-03-26T04:33:05.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:05 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:33:05.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:05 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:33:05.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:05 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "hit_set_count","val": "2"}]': finished 2024-03-26T04:33:06.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:06 smithi183 ceph-mon[33745]: pgmap v1097: 268 pgs: 268 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-26T04:33:06.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:06 smithi183 ceph-mon[33745]: osdmap e708: 8 total, 8 up, 8 in 2024-03-26T04:33:06.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:06 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:33:06.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:06 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:33:06.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:06 smithi183 ceph-mon[33745]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:33:06.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:06 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:33:06.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:06 smithi183 ceph-mon[33745]: osdmap e709: 8 total, 8 up, 8 in 2024-03-26T04:33:06.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:06 smithi016 ceph-mon[29101]: pgmap v1097: 268 pgs: 268 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-26T04:33:06.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:06 smithi016 ceph-mon[29101]: osdmap e708: 8 total, 8 up, 8 in 2024-03-26T04:33:06.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:06 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:33:06.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:06 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:33:06.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:06 smithi016 ceph-mon[29101]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:33:06.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:06 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:33:06.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:06 smithi016 ceph-mon[29101]: osdmap e709: 8 total, 8 up, 8 in 2024-03-26T04:33:06.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:06 smithi016 ceph-mon[37215]: pgmap v1097: 268 pgs: 268 active+clean; 4.4 MiB data, 684 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2024-03-26T04:33:06.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:06 smithi016 ceph-mon[37215]: osdmap e708: 8 total, 8 up, 8 in 2024-03-26T04:33:06.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:06 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:33:06.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:06 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:33:06.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:06 smithi016 ceph-mon[37215]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:33:06.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:06 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:33:06.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:06 smithi016 ceph-mon[37215]: osdmap e709: 8 total, 8 up, 8 in 2024-03-26T04:33:07.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:07 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:33:07.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:07 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:33:07.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:07 smithi183 ceph-mon[33745]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:33:07.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:07 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T04:33:07.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:07 smithi183 ceph-mon[33745]: osdmap e710: 8 total, 8 up, 8 in 2024-03-26T04:33:07.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:07 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:33:07.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:07 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:33:07.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:07 smithi016 ceph-mon[29101]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:33:07.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:07 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T04:33:07.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:07 smithi016 ceph-mon[29101]: osdmap e710: 8 total, 8 up, 8 in 2024-03-26T04:33:07.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:07 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:33:07.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:07 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:33:07.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:07 smithi016 ceph-mon[37215]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:33:07.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:07 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T04:33:07.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:07 smithi016 ceph-mon[37215]: osdmap e710: 8 total, 8 up, 8 in 2024-03-26T04:33:08.745 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:08 smithi016 ceph-mon[29101]: pgmap v1100: 268 pgs: 268 active+clean; 4.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:33:08.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:08 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T04:33:08.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:08 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T04:33:08.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:08 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:33:08.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:08 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T04:33:08.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:08 smithi016 ceph-mon[29101]: osdmap e711: 8 total, 8 up, 8 in 2024-03-26T04:33:08.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:08 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T04:33:08.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:08 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T04:33:08.746 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:08 smithi016 ceph-mon[37215]: pgmap v1100: 268 pgs: 268 active+clean; 4.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:33:08.746 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:08 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T04:33:08.746 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:08 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T04:33:08.747 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:08 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:33:08.747 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:08 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T04:33:08.747 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:08 smithi016 ceph-mon[37215]: osdmap e711: 8 total, 8 up, 8 in 2024-03-26T04:33:08.747 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:08 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T04:33:08.747 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:08 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T04:33:08.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:08 smithi183 ceph-mon[33745]: pgmap v1100: 268 pgs: 268 active+clean; 4.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:33:08.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:08 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T04:33:08.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:08 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T04:33:08.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:08 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:33:08.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:08 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T04:33:08.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:08 smithi183 ceph-mon[33745]: osdmap e711: 8 total, 8 up, 8 in 2024-03-26T04:33:08.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:08 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T04:33:08.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:08 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "hit_set_grade_decay_rate","val": "20"}]: dispatch 2024-03-26T04:33:09.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:33:08 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:33:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:33:10.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:10 smithi183 ceph-mon[33745]: pgmap v1103: 268 pgs: 268 active+clean; 4.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:33:10.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:10 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-26T04:33:10.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:10 smithi183 ceph-mon[33745]: osdmap e712: 8 total, 8 up, 8 in 2024-03-26T04:33:10.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:10 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T04:33:10.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:10 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T04:33:10.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:10 smithi016 ceph-mon[37215]: pgmap v1103: 268 pgs: 268 active+clean; 4.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:33:10.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:10 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-26T04:33:10.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:10 smithi016 ceph-mon[37215]: osdmap e712: 8 total, 8 up, 8 in 2024-03-26T04:33:10.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:10 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T04:33:10.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:10 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T04:33:10.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:10 smithi016 ceph-mon[29101]: pgmap v1103: 268 pgs: 268 active+clean; 4.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:33:10.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:10 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "hit_set_grade_decay_rate","val": "20"}]': finished 2024-03-26T04:33:10.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:10 smithi016 ceph-mon[29101]: osdmap e712: 8 total, 8 up, 8 in 2024-03-26T04:33:10.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:10 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T04:33:10.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:10 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "hit_set_search_last_n","val": "1"}]: dispatch 2024-03-26T04:33:11.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:11 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-26T04:33:11.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:11 smithi183 ceph-mon[33745]: osdmap e713: 8 total, 8 up, 8 in 2024-03-26T04:33:11.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:11 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:33:11.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:11 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:33:11.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:11 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-26T04:33:11.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:11 smithi016 ceph-mon[37215]: osdmap e713: 8 total, 8 up, 8 in 2024-03-26T04:33:11.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:11 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:33:11.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:11 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:33:11.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:11 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-142","var": "hit_set_search_last_n","val": "1"}]': finished 2024-03-26T04:33:11.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:11 smithi016 ceph-mon[29101]: osdmap e713: 8 total, 8 up, 8 in 2024-03-26T04:33:11.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:11 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:33:11.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:11 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:33:12.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:12 smithi183 ceph-mon[33745]: pgmap v1106: 268 pgs: 268 active+clean; 4.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:33:12.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:12 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:33:12.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:12 smithi183 ceph-mon[33745]: osdmap e714: 8 total, 8 up, 8 in 2024-03-26T04:33:12.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:12 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-142"}]: dispatch 2024-03-26T04:33:12.738 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:12 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-142"}]: dispatch 2024-03-26T04:33:12.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:12 smithi016 ceph-mon[29101]: pgmap v1106: 268 pgs: 268 active+clean; 4.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:33:12.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:12 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:33:12.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:12 smithi016 ceph-mon[29101]: osdmap e714: 8 total, 8 up, 8 in 2024-03-26T04:33:12.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:12 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-142"}]: dispatch 2024-03-26T04:33:12.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:12 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-142"}]: dispatch 2024-03-26T04:33:12.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:12 smithi016 ceph-mon[37215]: pgmap v1106: 268 pgs: 268 active+clean; 4.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:33:12.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:12 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:33:12.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:12 smithi016 ceph-mon[37215]: osdmap e714: 8 total, 8 up, 8 in 2024-03-26T04:33:12.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:12 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-142"}]: dispatch 2024-03-26T04:33:12.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:12 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-142"}]: dispatch 2024-03-26T04:33:13.041 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:33:12 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:33:13.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:13 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-142"}]': finished 2024-03-26T04:33:13.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:13 smithi183 ceph-mon[33745]: osdmap e715: 8 total, 8 up, 8 in 2024-03-26T04:33:13.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:13 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:33:13.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:13 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:33:13.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:13 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-142"}]: dispatch 2024-03-26T04:33:13.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:13 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-142"}]: dispatch 2024-03-26T04:33:13.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:13 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-142"}]': finished 2024-03-26T04:33:13.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:13 smithi016 ceph-mon[29101]: osdmap e715: 8 total, 8 up, 8 in 2024-03-26T04:33:13.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:13 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:33:13.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:13 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:33:13.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:13 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-142"}]: dispatch 2024-03-26T04:33:13.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:13 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-142"}]: dispatch 2024-03-26T04:33:13.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:13 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-142"}]': finished 2024-03-26T04:33:13.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:13 smithi016 ceph-mon[37215]: osdmap e715: 8 total, 8 up, 8 in 2024-03-26T04:33:13.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:13 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:33:13.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:13 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:33:13.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:13 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-142"}]: dispatch 2024-03-26T04:33:13.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:13 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-142"}]: dispatch 2024-03-26T04:33:14.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:14 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:33:14.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:14 smithi183 ceph-mon[33745]: pgmap v1109: 268 pgs: 268 active+clean; 4.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T04:33:14.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:14 smithi183 ceph-mon[33745]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:33:14.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:14 smithi183 ceph-mon[33745]: osdmap e716: 8 total, 8 up, 8 in 2024-03-26T04:33:14.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:14 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:33:14.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:14 smithi016 ceph-mon[37215]: pgmap v1109: 268 pgs: 268 active+clean; 4.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T04:33:14.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:14 smithi016 ceph-mon[37215]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:33:14.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:14 smithi016 ceph-mon[37215]: osdmap e716: 8 total, 8 up, 8 in 2024-03-26T04:33:14.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:14 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:33:14.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:14 smithi016 ceph-mon[29101]: pgmap v1109: 268 pgs: 268 active+clean; 4.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T04:33:14.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:14 smithi016 ceph-mon[29101]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:33:14.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:14 smithi016 ceph-mon[29101]: osdmap e716: 8 total, 8 up, 8 in 2024-03-26T04:33:15.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:15 smithi183 ceph-mon[33745]: osdmap e717: 8 total, 8 up, 8 in 2024-03-26T04:33:15.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:15 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:33:15.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:15 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:33:15.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:15 smithi016 ceph-mon[29101]: osdmap e717: 8 total, 8 up, 8 in 2024-03-26T04:33:15.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:15 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:33:15.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:15 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:33:15.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:15 smithi016 ceph-mon[37215]: osdmap e717: 8 total, 8 up, 8 in 2024-03-26T04:33:15.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:15 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:33:15.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:15 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-144","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:33:16.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:16 smithi183 ceph-mon[33745]: pgmap v1112: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T04:33:16.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:16 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:33:16.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:16 smithi183 ceph-mon[33745]: osdmap e718: 8 total, 8 up, 8 in 2024-03-26T04:33:16.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:16 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:33:16.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:16 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:33:16.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:16 smithi016 ceph-mon[29101]: pgmap v1112: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T04:33:16.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:16 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:33:16.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:16 smithi016 ceph-mon[29101]: osdmap e718: 8 total, 8 up, 8 in 2024-03-26T04:33:16.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:16 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:33:16.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:16 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:33:16.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:16 smithi016 ceph-mon[37215]: pgmap v1112: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-26T04:33:16.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:16 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-144","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:33:16.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:16 smithi016 ceph-mon[37215]: osdmap e718: 8 total, 8 up, 8 in 2024-03-26T04:33:16.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:16 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:33:16.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:16 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-144", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:33:17.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:17 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-144", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:33:17.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:17 smithi183 ceph-mon[33745]: osdmap e719: 8 total, 8 up, 8 in 2024-03-26T04:33:17.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:17 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-144"}]: dispatch 2024-03-26T04:33:17.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:17 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-144"}]: dispatch 2024-03-26T04:33:17.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:17 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-144", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:33:17.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:17 smithi016 ceph-mon[37215]: osdmap e719: 8 total, 8 up, 8 in 2024-03-26T04:33:17.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:17 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-144"}]: dispatch 2024-03-26T04:33:17.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:17 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-144"}]: dispatch 2024-03-26T04:33:17.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:17 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-144", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:33:17.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:17 smithi016 ceph-mon[29101]: osdmap e719: 8 total, 8 up, 8 in 2024-03-26T04:33:17.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:17 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-144"}]: dispatch 2024-03-26T04:33:17.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:17 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-144"}]: dispatch 2024-03-26T04:33:18.745 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:18 smithi016 ceph-mon[37215]: pgmap v1115: 268 pgs: 10 unknown, 258 active+clean; 4.4 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:33:18.745 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:18 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-144"}]': finished 2024-03-26T04:33:18.745 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:18 smithi016 ceph-mon[37215]: osdmap e720: 8 total, 8 up, 8 in 2024-03-26T04:33:18.745 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:18 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-144", "mode": "readproxy"}]: dispatch 2024-03-26T04:33:18.745 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:18 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-144", "mode": "readproxy"}]: dispatch 2024-03-26T04:33:18.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:18 smithi016 ceph-mon[29101]: pgmap v1115: 268 pgs: 10 unknown, 258 active+clean; 4.4 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:33:18.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:18 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-144"}]': finished 2024-03-26T04:33:18.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:18 smithi016 ceph-mon[29101]: osdmap e720: 8 total, 8 up, 8 in 2024-03-26T04:33:18.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:18 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-144", "mode": "readproxy"}]: dispatch 2024-03-26T04:33:18.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:18 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-144", "mode": "readproxy"}]: dispatch 2024-03-26T04:33:18.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:18 smithi183 ceph-mon[33745]: pgmap v1115: 268 pgs: 10 unknown, 258 active+clean; 4.4 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:33:18.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:18 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-144"}]': finished 2024-03-26T04:33:18.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:18 smithi183 ceph-mon[33745]: osdmap e720: 8 total, 8 up, 8 in 2024-03-26T04:33:18.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:18 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-144", "mode": "readproxy"}]: dispatch 2024-03-26T04:33:18.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:18 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-144", "mode": "readproxy"}]: dispatch 2024-03-26T04:33:19.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:33:18 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:33:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:33:19.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:19 smithi183 ceph-mon[33745]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:33:19.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:19 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-144", "mode": "readproxy"}]': finished 2024-03-26T04:33:19.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:19 smithi183 ceph-mon[33745]: osdmap e721: 8 total, 8 up, 8 in 2024-03-26T04:33:19.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:19 smithi016 ceph-mon[37215]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:33:19.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:19 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-144", "mode": "readproxy"}]': finished 2024-03-26T04:33:19.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:19 smithi016 ceph-mon[37215]: osdmap e721: 8 total, 8 up, 8 in 2024-03-26T04:33:19.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:19 smithi016 ceph-mon[29101]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:33:19.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:19 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-144", "mode": "readproxy"}]': finished 2024-03-26T04:33:19.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:19 smithi016 ceph-mon[29101]: osdmap e721: 8 total, 8 up, 8 in 2024-03-26T04:33:20.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:20 smithi183 ceph-mon[33745]: pgmap v1118: 268 pgs: 268 active+clean; 4.4 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:33:20.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:20 smithi016 ceph-mon[37215]: pgmap v1118: 268 pgs: 268 active+clean; 4.4 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:33:20.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:20 smithi016 ceph-mon[29101]: pgmap v1118: 268 pgs: 268 active+clean; 4.4 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 255 B/s wr, 1 op/s 2024-03-26T04:33:21.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:21 smithi183 ceph-mon[33745]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:33:21.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:21 smithi016 ceph-mon[37215]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:33:21.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:21 smithi016 ceph-mon[29101]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:33:22.750 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:22 smithi183 ceph-mon[33745]: pgmap v1119: 268 pgs: 268 active+clean; 4.4 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 881 B/s rd, 176 B/s wr, 1 op/s 2024-03-26T04:33:22.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:22 smithi016 ceph-mon[29101]: pgmap v1119: 268 pgs: 268 active+clean; 4.4 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 881 B/s rd, 176 B/s wr, 1 op/s 2024-03-26T04:33:22.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:22 smithi016 ceph-mon[37215]: pgmap v1119: 268 pgs: 268 active+clean; 4.4 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 881 B/s rd, 176 B/s wr, 1 op/s 2024-03-26T04:33:23.041 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:33:22 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:33:24.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:23 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:33:24.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:23 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:33:24.294 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:23 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:33:24.295 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:23 smithi183 ceph-mon[33745]: pgmap v1120: 268 pgs: 268 active+clean; 4.4 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 150 B/s wr, 1 op/s 2024-03-26T04:33:24.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:23 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:33:24.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:23 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:33:24.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:23 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:33:24.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:23 smithi016 ceph-mon[37215]: pgmap v1120: 268 pgs: 268 active+clean; 4.4 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 150 B/s wr, 1 op/s 2024-03-26T04:33:24.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:23 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:33:24.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:23 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:33:24.412 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:23 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:33:24.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:23 smithi016 ceph-mon[29101]: pgmap v1120: 268 pgs: 268 active+clean; 4.4 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 150 B/s wr, 1 op/s 2024-03-26T04:33:26.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:26 smithi016 ceph-mon[29101]: pgmap v1121: 268 pgs: 268 active+clean; 4.4 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T04:33:26.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:26 smithi016 ceph-mon[37215]: pgmap v1121: 268 pgs: 268 active+clean; 4.4 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T04:33:26.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:26 smithi183 ceph-mon[33745]: pgmap v1121: 268 pgs: 268 active+clean; 4.4 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T04:33:28.594 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:28 smithi183 ceph-mon[33745]: pgmap v1122: 268 pgs: 268 active+clean; 4.4 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:33:28.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:28 smithi016 ceph-mon[29101]: pgmap v1122: 268 pgs: 268 active+clean; 4.4 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:33:28.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:28 smithi016 ceph-mon[37215]: pgmap v1122: 268 pgs: 268 active+clean; 4.4 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:33:29.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:33:28 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:33:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:33:29.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:29 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:33:29.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:29 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:33:29.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:29 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:33:29.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:29 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:33:29.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:29 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:33:29.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:29 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:33:30.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:30 smithi016 ceph-mon[29101]: pgmap v1123: 268 pgs: 268 active+clean; 4.4 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T04:33:30.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:30 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:33:30.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:30 smithi016 ceph-mon[29101]: osdmap e722: 8 total, 8 up, 8 in 2024-03-26T04:33:30.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:30 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-144"}]: dispatch 2024-03-26T04:33:30.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:30 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-144"}]: dispatch 2024-03-26T04:33:30.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:30 smithi016 ceph-mon[37215]: pgmap v1123: 268 pgs: 268 active+clean; 4.4 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T04:33:30.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:30 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:33:30.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:30 smithi016 ceph-mon[37215]: osdmap e722: 8 total, 8 up, 8 in 2024-03-26T04:33:30.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:30 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-144"}]: dispatch 2024-03-26T04:33:30.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:30 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-144"}]: dispatch 2024-03-26T04:33:30.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:30 smithi183 ceph-mon[33745]: pgmap v1123: 268 pgs: 268 active+clean; 4.4 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-26T04:33:30.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:30 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:33:30.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:30 smithi183 ceph-mon[33745]: osdmap e722: 8 total, 8 up, 8 in 2024-03-26T04:33:30.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:30 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-144"}]: dispatch 2024-03-26T04:33:30.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:30 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-144"}]: dispatch 2024-03-26T04:33:31.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:31 smithi016 ceph-mon[29101]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:33:31.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:31 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-144"}]': finished 2024-03-26T04:33:31.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:31 smithi016 ceph-mon[29101]: osdmap e723: 8 total, 8 up, 8 in 2024-03-26T04:33:31.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:31 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:33:31.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:31 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:33:31.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:31 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-144"}]: dispatch 2024-03-26T04:33:31.666 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:31 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-144"}]: dispatch 2024-03-26T04:33:31.666 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:31 smithi016 ceph-mon[37215]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:33:31.666 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:31 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-144"}]': finished 2024-03-26T04:33:31.666 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:31 smithi016 ceph-mon[37215]: osdmap e723: 8 total, 8 up, 8 in 2024-03-26T04:33:31.667 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:31 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:33:31.667 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:31 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:33:31.667 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:31 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-144"}]: dispatch 2024-03-26T04:33:31.667 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:31 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-144"}]: dispatch 2024-03-26T04:33:31.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:31 smithi183 ceph-mon[33745]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:33:31.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:31 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-144"}]': finished 2024-03-26T04:33:31.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:31 smithi183 ceph-mon[33745]: osdmap e723: 8 total, 8 up, 8 in 2024-03-26T04:33:31.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:31 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:33:31.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:31 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:33:31.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:31 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-144"}]: dispatch 2024-03-26T04:33:31.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:31 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-144"}]: dispatch 2024-03-26T04:33:32.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:32 smithi016 ceph-mon[29101]: pgmap v1126: 268 pgs: 268 active+clean; 4.4 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:33:32.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:32 smithi016 ceph-mon[29101]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:33:32.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:32 smithi016 ceph-mon[29101]: osdmap e724: 8 total, 8 up, 8 in 2024-03-26T04:33:32.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:32 smithi016 ceph-mon[37215]: pgmap v1126: 268 pgs: 268 active+clean; 4.4 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:33:32.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:32 smithi016 ceph-mon[37215]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:33:32.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:32 smithi016 ceph-mon[37215]: osdmap e724: 8 total, 8 up, 8 in 2024-03-26T04:33:32.760 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:32 smithi183 ceph-mon[33745]: pgmap v1126: 268 pgs: 268 active+clean; 4.4 MiB data, 686 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:33:32.761 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:32 smithi183 ceph-mon[33745]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:33:32.761 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:32 smithi183 ceph-mon[33745]: osdmap e724: 8 total, 8 up, 8 in 2024-03-26T04:33:33.041 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:33:32 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:33:33.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:33 smithi016 ceph-mon[29101]: osdmap e725: 8 total, 8 up, 8 in 2024-03-26T04:33:33.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:33 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:33:33.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:33 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:33:33.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:33 smithi016 ceph-mon[37215]: osdmap e725: 8 total, 8 up, 8 in 2024-03-26T04:33:33.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:33 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:33:33.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:33 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:33:33.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:33 smithi183 ceph-mon[33745]: osdmap e725: 8 total, 8 up, 8 in 2024-03-26T04:33:33.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:33 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:33:33.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:33 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-146","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:33:34.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:34 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:33:34.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:34 smithi016 ceph-mon[37215]: pgmap v1129: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:33:34.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:34 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:33:34.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:34 smithi016 ceph-mon[37215]: osdmap e726: 8 total, 8 up, 8 in 2024-03-26T04:33:34.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:34 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:33:34.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:34 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:33:34.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:34 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:33:34.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:34 smithi016 ceph-mon[29101]: pgmap v1129: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:33:34.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:34 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:33:34.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:34 smithi016 ceph-mon[29101]: osdmap e726: 8 total, 8 up, 8 in 2024-03-26T04:33:34.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:34 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:33:34.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:34 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:33:34.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:34 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:33:34.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:34 smithi183 ceph-mon[33745]: pgmap v1129: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:33:34.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:34 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-146","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:33:34.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:34 smithi183 ceph-mon[33745]: osdmap e726: 8 total, 8 up, 8 in 2024-03-26T04:33:34.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:34 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:33:34.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:34 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-146", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:33:35.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:35 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-146", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:33:35.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:35 smithi016 ceph-mon[37215]: osdmap e727: 8 total, 8 up, 8 in 2024-03-26T04:33:35.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:35 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-146"}]: dispatch 2024-03-26T04:33:35.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:35 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-146"}]: dispatch 2024-03-26T04:33:35.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:35 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-146"}]': finished 2024-03-26T04:33:35.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:35 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-146", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:33:35.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:35 smithi016 ceph-mon[29101]: osdmap e727: 8 total, 8 up, 8 in 2024-03-26T04:33:35.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:35 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-146"}]: dispatch 2024-03-26T04:33:35.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:35 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-146"}]: dispatch 2024-03-26T04:33:35.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:35 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-146"}]': finished 2024-03-26T04:33:35.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:35 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-146", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:33:35.816 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:35 smithi183 ceph-mon[33745]: osdmap e727: 8 total, 8 up, 8 in 2024-03-26T04:33:35.817 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:35 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-146"}]: dispatch 2024-03-26T04:33:35.817 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:35 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-146"}]: dispatch 2024-03-26T04:33:35.817 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:35 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi016-74534-111", "overlaypool": "test-rados-api-smithi016-74534-146"}]': finished 2024-03-26T04:33:36.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:36 smithi016 ceph-mon[29101]: pgmap v1132: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:33:36.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:36 smithi016 ceph-mon[29101]: osdmap e728: 8 total, 8 up, 8 in 2024-03-26T04:33:36.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:36 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-146", "mode": "writeback"}]: dispatch 2024-03-26T04:33:36.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:36 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-146", "mode": "writeback"}]: dispatch 2024-03-26T04:33:36.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:36 smithi016 ceph-mon[29101]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:33:36.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:36 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-146", "mode": "writeback"}]': finished 2024-03-26T04:33:36.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:36 smithi016 ceph-mon[29101]: osdmap e729: 8 total, 8 up, 8 in 2024-03-26T04:33:36.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:36 smithi016 ceph-mon[37215]: pgmap v1132: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:33:36.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:36 smithi016 ceph-mon[37215]: osdmap e728: 8 total, 8 up, 8 in 2024-03-26T04:33:36.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:36 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-146", "mode": "writeback"}]: dispatch 2024-03-26T04:33:36.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:36 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-146", "mode": "writeback"}]: dispatch 2024-03-26T04:33:36.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:36 smithi016 ceph-mon[37215]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:33:36.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:36 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-146", "mode": "writeback"}]': finished 2024-03-26T04:33:36.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:36 smithi016 ceph-mon[37215]: osdmap e729: 8 total, 8 up, 8 in 2024-03-26T04:33:36.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:36 smithi183 ceph-mon[33745]: pgmap v1132: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:33:36.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:36 smithi183 ceph-mon[33745]: osdmap e728: 8 total, 8 up, 8 in 2024-03-26T04:33:36.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:36 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-146", "mode": "writeback"}]: dispatch 2024-03-26T04:33:36.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:36 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-146", "mode": "writeback"}]: dispatch 2024-03-26T04:33:36.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:36 smithi183 ceph-mon[33745]: Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-03-26T04:33:36.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:36 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi016-74534-146", "mode": "writeback"}]': finished 2024-03-26T04:33:36.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:36 smithi183 ceph-mon[33745]: osdmap e729: 8 total, 8 up, 8 in 2024-03-26T04:33:37.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:37 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:33:37.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:37 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:33:37.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:37 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:33:37.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:37 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:33:37.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:37 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:33:37.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:37 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-146","var": "hit_set_count","val": "2"}]: dispatch 2024-03-26T04:33:38.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:38 smithi016 ceph-mon[37215]: pgmap v1135: 268 pgs: 8 unknown, 260 active+clean; 4.4 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-26T04:33:38.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:38 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-146","var": "hit_set_count","val": "2"}]': finished 2024-03-26T04:33:38.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:38 smithi016 ceph-mon[37215]: osdmap e730: 8 total, 8 up, 8 in 2024-03-26T04:33:38.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:38 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:33:38.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:38 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:33:38.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:38 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:33:38.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:38 smithi016 ceph-mon[29101]: pgmap v1135: 268 pgs: 8 unknown, 260 active+clean; 4.4 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-26T04:33:38.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:38 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-146","var": "hit_set_count","val": "2"}]': finished 2024-03-26T04:33:38.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:38 smithi016 ceph-mon[29101]: osdmap e730: 8 total, 8 up, 8 in 2024-03-26T04:33:38.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:38 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:33:38.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:38 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:33:38.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:38 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:33:38.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:38 smithi183 ceph-mon[33745]: pgmap v1135: 268 pgs: 8 unknown, 260 active+clean; 4.4 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-26T04:33:38.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:38 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-146","var": "hit_set_count","val": "2"}]': finished 2024-03-26T04:33:38.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:38 smithi183 ceph-mon[33745]: osdmap e730: 8 total, 8 up, 8 in 2024-03-26T04:33:38.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:38 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:33:38.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:38 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-146","var": "hit_set_period","val": "600"}]: dispatch 2024-03-26T04:33:38.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:38 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:33:39.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:33:38 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:33:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:33:39.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:39 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-146","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:33:39.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:39 smithi016 ceph-mon[29101]: osdmap e731: 8 total, 8 up, 8 in 2024-03-26T04:33:39.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:39 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:33:39.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:39 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:33:39.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:39 smithi016 ceph-mon[29101]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:33:39.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:39 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-146","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T04:33:39.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:39 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-146","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:33:39.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:39 smithi016 ceph-mon[37215]: osdmap e731: 8 total, 8 up, 8 in 2024-03-26T04:33:39.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:39 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:33:39.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:39 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:33:39.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:39 smithi016 ceph-mon[37215]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:33:39.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:39 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-146","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T04:33:39.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:39 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-146","var": "hit_set_period","val": "600"}]': finished 2024-03-26T04:33:39.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:39 smithi183 ceph-mon[33745]: osdmap e731: 8 total, 8 up, 8 in 2024-03-26T04:33:39.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:39 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:33:39.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:39 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-146","var": "hit_set_type","val": "bloom"}]: dispatch 2024-03-26T04:33:39.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:39 smithi183 ceph-mon[33745]: Health check cleared: CACHE_POOL_NO_HIT_SET (was: 1 cache pools are missing hit_sets) 2024-03-26T04:33:39.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:39 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-146","var": "hit_set_type","val": "bloom"}]': finished 2024-03-26T04:33:40.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:40 smithi016 ceph-mon[29101]: pgmap v1138: 268 pgs: 268 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-26T04:33:40.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:40 smithi016 ceph-mon[29101]: osdmap e732: 8 total, 8 up, 8 in 2024-03-26T04:33:40.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:40 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T04:33:40.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:40 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T04:33:40.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:40 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T04:33:40.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:40 smithi016 ceph-mon[29101]: osdmap e733: 8 total, 8 up, 8 in 2024-03-26T04:33:40.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:40 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T04:33:40.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:40 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T04:33:40.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:40 smithi016 ceph-mon[37215]: pgmap v1138: 268 pgs: 268 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-26T04:33:40.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:40 smithi016 ceph-mon[37215]: osdmap e732: 8 total, 8 up, 8 in 2024-03-26T04:33:40.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:40 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T04:33:40.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:40 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T04:33:40.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:40 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T04:33:40.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:40 smithi016 ceph-mon[37215]: osdmap e733: 8 total, 8 up, 8 in 2024-03-26T04:33:40.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:40 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T04:33:40.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:40 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T04:33:40.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:40 smithi183 ceph-mon[33745]: pgmap v1138: 268 pgs: 268 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 1023 B/s wr, 4 op/s 2024-03-26T04:33:40.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:40 smithi183 ceph-mon[33745]: osdmap e732: 8 total, 8 up, 8 in 2024-03-26T04:33:40.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:40 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T04:33:40.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:40 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-146","var": "min_read_recency_for_promote","val": "1"}]: dispatch 2024-03-26T04:33:40.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:40 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-146","var": "min_read_recency_for_promote","val": "1"}]': finished 2024-03-26T04:33:40.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:40 smithi183 ceph-mon[33745]: osdmap e733: 8 total, 8 up, 8 in 2024-03-26T04:33:40.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:40 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T04:33:40.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:40 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-146","var": "target_max_objects","val": "1"}]: dispatch 2024-03-26T04:33:41.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:41 smithi016 ceph-mon[37215]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:33:41.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:41 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-146","var": "target_max_objects","val": "1"}]': finished 2024-03-26T04:33:41.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:41 smithi016 ceph-mon[37215]: osdmap e734: 8 total, 8 up, 8 in 2024-03-26T04:33:41.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:41 smithi016 ceph-mon[29101]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:33:41.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:41 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-146","var": "target_max_objects","val": "1"}]': finished 2024-03-26T04:33:41.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:41 smithi016 ceph-mon[29101]: osdmap e734: 8 total, 8 up, 8 in 2024-03-26T04:33:41.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:41 smithi183 ceph-mon[33745]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:33:41.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:41 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-146","var": "target_max_objects","val": "1"}]': finished 2024-03-26T04:33:41.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:41 smithi183 ceph-mon[33745]: osdmap e734: 8 total, 8 up, 8 in 2024-03-26T04:33:42.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:42 smithi016 ceph-mon[29101]: pgmap v1141: 268 pgs: 268 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:33:42.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:42 smithi016 ceph-mon[37215]: pgmap v1141: 268 pgs: 268 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:33:42.765 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:42 smithi183 ceph-mon[33745]: pgmap v1141: 268 pgs: 268 active+clean; 4.4 MiB data, 709 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 1 op/s 2024-03-26T04:33:43.041 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:33:42 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:33:43.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:43 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:33:43.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:43 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:33:43.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:43 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:33:44.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:44 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:33:44.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:44 smithi016 ceph-mon[29101]: pgmap v1143: 268 pgs: 268 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:33:44.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:44 smithi016 ceph-mon[29101]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-26T04:33:44.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:44 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:33:44.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:44 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:33:44.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:44 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:33:44.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:44 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:33:44.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:44 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:33:44.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:44 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:33:44.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:44 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:33:44.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:44 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:33:44.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:44 smithi016 ceph-mon[37215]: pgmap v1143: 268 pgs: 268 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:33:44.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:44 smithi016 ceph-mon[37215]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-26T04:33:44.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:44 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:33:44.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:44 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:33:44.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:44 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:33:44.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:44 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:33:44.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:44 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:33:44.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:44 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:33:44.665 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:44 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:33:44.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:44 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:33:44.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:44 smithi183 ceph-mon[33745]: pgmap v1143: 268 pgs: 268 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:33:44.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:44 smithi183 ceph-mon[33745]: Health check failed: 1 cache pools at or near target size (CACHE_POOL_NEAR_FULL) 2024-03-26T04:33:44.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:44 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:33:44.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:44 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:33:44.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:44 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:33:44.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:44 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:33:44.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:44 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:33:44.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:44 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:33:44.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:44 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:33:46.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:46 smithi016 ceph-mon[29101]: pgmap v1144: 268 pgs: 268 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:33:46.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:46 smithi016 ceph-mon[37215]: pgmap v1144: 268 pgs: 268 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:33:46.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:46 smithi183 ceph-mon[33745]: pgmap v1144: 268 pgs: 268 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:33:48.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:48 smithi016 ceph-mon[29101]: pgmap v1145: 268 pgs: 268 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:33:48.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:48 smithi016 ceph-mon[37215]: pgmap v1145: 268 pgs: 268 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:33:48.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:48 smithi183 ceph-mon[33745]: pgmap v1145: 268 pgs: 268 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:33:49.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:33:48 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:33:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:33:50.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:50 smithi016 ceph-mon[29101]: pgmap v1146: 268 pgs: 268 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:33:50.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:50 smithi016 ceph-mon[37215]: pgmap v1146: 268 pgs: 268 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:33:50.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:50 smithi183 ceph-mon[33745]: pgmap v1146: 268 pgs: 268 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:33:51.541 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:33:51 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=sqlstore.transactions t=2024-03-26T04:33:51.1253145Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-26T04:33:52.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:52 smithi183 ceph-mon[33745]: pgmap v1147: 268 pgs: 268 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:33:52.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:52 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:33:52.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:52 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:33:52.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:52 smithi016 ceph-mon[37215]: pgmap v1147: 268 pgs: 268 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:33:52.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:52 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:33:52.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:52 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:33:52.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:52 smithi016 ceph-mon[29101]: pgmap v1147: 268 pgs: 268 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1 op/s 2024-03-26T04:33:52.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:52 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:33:52.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:52 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:33:53.041 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:33:52 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:33:53.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:53 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:33:53.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:53 smithi183 ceph-mon[33745]: osdmap e735: 8 total, 8 up, 8 in 2024-03-26T04:33:53.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:53 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-146"}]: dispatch 2024-03-26T04:33:53.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:53 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-146"}]: dispatch 2024-03-26T04:33:53.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:53 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:33:53.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:53 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:33:53.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:53 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:33:53.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:53 smithi016 ceph-mon[37215]: osdmap e735: 8 total, 8 up, 8 in 2024-03-26T04:33:53.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:53 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-146"}]: dispatch 2024-03-26T04:33:53.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:53 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-146"}]: dispatch 2024-03-26T04:33:53.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:53 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:33:53.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:53 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:33:53.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:53 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:33:53.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:53 smithi016 ceph-mon[29101]: osdmap e735: 8 total, 8 up, 8 in 2024-03-26T04:33:53.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:53 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-146"}]: dispatch 2024-03-26T04:33:53.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:53 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-146"}]: dispatch 2024-03-26T04:33:53.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:53 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:33:53.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:53 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:33:54.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:54 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:33:54.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:54 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-146"}]': finished 2024-03-26T04:33:54.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:54 smithi183 ceph-mon[33745]: osdmap e736: 8 total, 8 up, 8 in 2024-03-26T04:33:54.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:54 smithi183 ceph-mon[33745]: pgmap v1150: 268 pgs: 268 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T04:33:54.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:54 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:33:54.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:54 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:33:54.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:54 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-146"}]: dispatch 2024-03-26T04:33:54.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:54 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-146"}]: dispatch 2024-03-26T04:33:54.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:54 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:33:54.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:54 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-146"}]': finished 2024-03-26T04:33:54.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:54 smithi016 ceph-mon[37215]: osdmap e736: 8 total, 8 up, 8 in 2024-03-26T04:33:54.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:54 smithi016 ceph-mon[37215]: pgmap v1150: 268 pgs: 268 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T04:33:54.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:54 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:33:54.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:54 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:33:54.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:54 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-146"}]: dispatch 2024-03-26T04:33:54.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:54 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-146"}]: dispatch 2024-03-26T04:33:54.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:54 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:33:54.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:54 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-146"}]': finished 2024-03-26T04:33:54.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:54 smithi016 ceph-mon[29101]: osdmap e736: 8 total, 8 up, 8 in 2024-03-26T04:33:54.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:54 smithi016 ceph-mon[29101]: pgmap v1150: 268 pgs: 268 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-26T04:33:54.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:54 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:33:54.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:54 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:33:54.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:54 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-146"}]: dispatch 2024-03-26T04:33:54.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:54 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-146"}]: dispatch 2024-03-26T04:33:55.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:55 smithi183 ceph-mon[33745]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:33:55.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:55 smithi183 ceph-mon[33745]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-26T04:33:55.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:55 smithi183 ceph-mon[33745]: osdmap e737: 8 total, 8 up, 8 in 2024-03-26T04:33:55.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:55 smithi016 ceph-mon[29101]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:33:55.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:55 smithi016 ceph-mon[29101]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-26T04:33:55.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:55 smithi016 ceph-mon[29101]: osdmap e737: 8 total, 8 up, 8 in 2024-03-26T04:33:55.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:55 smithi016 ceph-mon[37215]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:33:55.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:55 smithi016 ceph-mon[37215]: Health check cleared: CACHE_POOL_NEAR_FULL (was: 1 cache pools at or near target size) 2024-03-26T04:33:55.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:55 smithi016 ceph-mon[37215]: osdmap e737: 8 total, 8 up, 8 in 2024-03-26T04:33:56.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:56 smithi183 ceph-mon[33745]: osdmap e738: 8 total, 8 up, 8 in 2024-03-26T04:33:56.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:56 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:33:56.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:56 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:33:56.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:56 smithi183 ceph-mon[33745]: pgmap v1153: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T04:33:56.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:56 smithi016 ceph-mon[37215]: osdmap e738: 8 total, 8 up, 8 in 2024-03-26T04:33:56.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:56 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:33:56.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:56 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:33:56.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:56 smithi016 ceph-mon[37215]: pgmap v1153: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T04:33:56.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:56 smithi016 ceph-mon[29101]: osdmap e738: 8 total, 8 up, 8 in 2024-03-26T04:33:56.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:56 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:33:56.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:56 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-148","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:33:56.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:56 smithi016 ceph-mon[29101]: pgmap v1153: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-03-26T04:33:57.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:57 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:33:57.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:57 smithi183 ceph-mon[33745]: osdmap e739: 8 total, 8 up, 8 in 2024-03-26T04:33:57.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:57 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:33:57.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:57 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:33:57.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:57 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:33:57.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:57 smithi016 ceph-mon[29101]: osdmap e739: 8 total, 8 up, 8 in 2024-03-26T04:33:57.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:57 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:33:57.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:57 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:33:57.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:57 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-148","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:33:57.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:57 smithi016 ceph-mon[37215]: osdmap e739: 8 total, 8 up, 8 in 2024-03-26T04:33:57.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:57 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:33:57.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:57 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-148", "force_nonempty": "--force-nonempty" }]: dispatch 2024-03-26T04:33:58.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:58 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-148", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:33:58.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:58 smithi183 ceph-mon[33745]: osdmap e740: 8 total, 8 up, 8 in 2024-03-26T04:33:58.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:58 smithi183 ceph-mon[33745]: pgmap v1156: 268 pgs: 11 unknown, 257 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:33:58.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-148"}]: dispatch 2024-03-26T04:33:58.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:58 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-148"}]: dispatch 2024-03-26T04:33:58.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:58 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-148", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:33:58.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:58 smithi016 ceph-mon[29101]: osdmap e740: 8 total, 8 up, 8 in 2024-03-26T04:33:58.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:58 smithi016 ceph-mon[29101]: pgmap v1156: 268 pgs: 11 unknown, 257 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:33:58.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-148"}]: dispatch 2024-03-26T04:33:58.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:58 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-148"}]: dispatch 2024-03-26T04:33:58.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:58 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-148", "force_nonempty": "--force-nonempty" }]': finished 2024-03-26T04:33:58.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:58 smithi016 ceph-mon[37215]: osdmap e740: 8 total, 8 up, 8 in 2024-03-26T04:33:58.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:58 smithi016 ceph-mon[37215]: pgmap v1156: 268 pgs: 11 unknown, 257 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:33:58.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-148"}]: dispatch 2024-03-26T04:33:58.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:58 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-148"}]: dispatch 2024-03-26T04:33:59.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:33:58 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:33:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:33:59.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:59 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-148"}]': finished 2024-03-26T04:33:59.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:59 smithi183 ceph-mon[33745]: osdmap e741: 8 total, 8 up, 8 in 2024-03-26T04:33:59.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:59 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:33:59.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:59 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:33:59.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:59 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-148"}]: dispatch 2024-03-26T04:33:59.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:33:59 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-148"}]: dispatch 2024-03-26T04:33:59.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:59 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-148"}]': finished 2024-03-26T04:33:59.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:59 smithi016 ceph-mon[29101]: osdmap e741: 8 total, 8 up, 8 in 2024-03-26T04:33:59.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:59 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:33:59.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:59 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:33:59.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:59 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-148"}]: dispatch 2024-03-26T04:33:59.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:33:59 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-148"}]: dispatch 2024-03-26T04:33:59.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:59 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-148"}]': finished 2024-03-26T04:33:59.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:59 smithi016 ceph-mon[37215]: osdmap e741: 8 total, 8 up, 8 in 2024-03-26T04:33:59.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:59 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:33:59.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:59 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:33:59.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:59 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-148"}]: dispatch 2024-03-26T04:33:59.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:33:59 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-148"}]: dispatch 2024-03-26T04:34:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:00 smithi183 ceph-mon[33745]: pgmap v1158: 268 pgs: 268 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 509 B/s wr, 1 op/s 2024-03-26T04:34:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:00 smithi183 ceph-mon[33745]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:34:00.542 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:00 smithi183 ceph-mon[33745]: osdmap e742: 8 total, 8 up, 8 in 2024-03-26T04:34:00.563 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:00 smithi016 ceph-mon[37215]: pgmap v1158: 268 pgs: 268 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 509 B/s wr, 1 op/s 2024-03-26T04:34:00.563 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:00 smithi016 ceph-mon[37215]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:34:00.563 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:00 smithi016 ceph-mon[37215]: osdmap e742: 8 total, 8 up, 8 in 2024-03-26T04:34:00.564 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:00 smithi016 ceph-mon[29101]: pgmap v1158: 268 pgs: 268 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 509 B/s wr, 1 op/s 2024-03-26T04:34:00.564 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:00 smithi016 ceph-mon[29101]: Health check update: 5 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:34:00.564 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:00 smithi016 ceph-mon[29101]: osdmap e742: 8 total, 8 up, 8 in 2024-03-26T04:34:01.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:01 smithi016 ceph-mon[37215]: osdmap e743: 8 total, 8 up, 8 in 2024-03-26T04:34:01.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:01 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:34:01.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:01 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:34:01.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:01 smithi016 ceph-mon[29101]: osdmap e743: 8 total, 8 up, 8 in 2024-03-26T04:34:01.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:01 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:34:01.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:01 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:34:01.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:01 smithi183 ceph-mon[33745]: osdmap e743: 8 total, 8 up, 8 in 2024-03-26T04:34:01.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:01 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:34:01.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:01 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-150","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:34:02.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:02 smithi016 ceph-mon[29101]: pgmap v1161: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:34:02.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:02 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:34:02.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:02 smithi016 ceph-mon[29101]: osdmap e744: 8 total, 8 up, 8 in 2024-03-26T04:34:02.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:02 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:34:02.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:02 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:34:02.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:02 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-150"}]: dispatch 2024-03-26T04:34:02.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:02 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-150"}]: dispatch 2024-03-26T04:34:02.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:02 smithi016 ceph-mon[37215]: pgmap v1161: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:34:02.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:02 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:34:02.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:02 smithi016 ceph-mon[37215]: osdmap e744: 8 total, 8 up, 8 in 2024-03-26T04:34:02.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:02 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:34:02.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:02 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:34:02.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:02 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-150"}]: dispatch 2024-03-26T04:34:02.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:02 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-150"}]: dispatch 2024-03-26T04:34:02.784 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:02 smithi183 ceph-mon[33745]: pgmap v1161: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 714 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:34:02.784 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:02 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-150","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:34:02.784 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:02 smithi183 ceph-mon[33745]: osdmap e744: 8 total, 8 up, 8 in 2024-03-26T04:34:02.784 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:02 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:34:02.784 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:02 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:34:02.784 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:02 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-150"}]: dispatch 2024-03-26T04:34:02.785 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:02 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-150"}]: dispatch 2024-03-26T04:34:03.041 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:34:02 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:34:03.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:03 smithi016 ceph-mon[29101]: osdmap e745: 8 total, 8 up, 8 in 2024-03-26T04:34:03.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:03 smithi016 ceph-mon[37215]: osdmap e745: 8 total, 8 up, 8 in 2024-03-26T04:34:03.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:03 smithi183 ceph-mon[33745]: osdmap e745: 8 total, 8 up, 8 in 2024-03-26T04:34:04.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:04 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:34:04.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:04 smithi016 ceph-mon[29101]: pgmap v1164: 236 pgs: 236 active+clean; 4.4 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:34:04.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:04 smithi016 ceph-mon[29101]: osdmap e746: 8 total, 8 up, 8 in 2024-03-26T04:34:04.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:04 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:34:04.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:04 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:34:04.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:04 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:34:04.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:04 smithi016 ceph-mon[37215]: pgmap v1164: 236 pgs: 236 active+clean; 4.4 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:34:04.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:04 smithi016 ceph-mon[37215]: osdmap e746: 8 total, 8 up, 8 in 2024-03-26T04:34:04.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:04 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:34:04.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:04 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:34:04.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:04 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:34:04.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:04 smithi183 ceph-mon[33745]: pgmap v1164: 236 pgs: 236 active+clean; 4.4 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-26T04:34:04.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:04 smithi183 ceph-mon[33745]: osdmap e746: 8 total, 8 up, 8 in 2024-03-26T04:34:04.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:04 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:34:04.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:04 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-152","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:34:05.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:05 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:34:05.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:05 smithi016 ceph-mon[37215]: osdmap e747: 8 total, 8 up, 8 in 2024-03-26T04:34:05.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:05 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:34:05.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:05 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:34:05.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:05 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-152"}]: dispatch 2024-03-26T04:34:05.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:05 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-152"}]: dispatch 2024-03-26T04:34:05.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:05 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:34:05.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:05 smithi016 ceph-mon[29101]: osdmap e747: 8 total, 8 up, 8 in 2024-03-26T04:34:05.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:05 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:34:05.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:05 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:34:05.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:05 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-152"}]: dispatch 2024-03-26T04:34:05.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:05 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-152"}]: dispatch 2024-03-26T04:34:05.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:05 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-152","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:34:05.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:05 smithi183 ceph-mon[33745]: osdmap e747: 8 total, 8 up, 8 in 2024-03-26T04:34:05.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:05 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:34:05.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:05 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:34:05.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:05 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-152"}]: dispatch 2024-03-26T04:34:05.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:05 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-152"}]: dispatch 2024-03-26T04:34:06.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:06 smithi016 ceph-mon[37215]: pgmap v1167: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T04:34:06.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:06 smithi016 ceph-mon[37215]: osdmap e748: 8 total, 8 up, 8 in 2024-03-26T04:34:06.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:06 smithi016 ceph-mon[37215]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:34:06.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:06 smithi016 ceph-mon[29101]: pgmap v1167: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T04:34:06.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:06 smithi016 ceph-mon[29101]: osdmap e748: 8 total, 8 up, 8 in 2024-03-26T04:34:06.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:06 smithi016 ceph-mon[29101]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:34:06.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:06 smithi183 ceph-mon[33745]: pgmap v1167: 268 pgs: 32 unknown, 236 active+clean; 4.4 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-26T04:34:06.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:06 smithi183 ceph-mon[33745]: osdmap e748: 8 total, 8 up, 8 in 2024-03-26T04:34:06.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:06 smithi183 ceph-mon[33745]: Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:34:07.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:07 smithi016 ceph-mon[29101]: osdmap e749: 8 total, 8 up, 8 in 2024-03-26T04:34:07.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:07 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:34:07.664 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:07 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:34:07.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:07 smithi016 ceph-mon[37215]: osdmap e749: 8 total, 8 up, 8 in 2024-03-26T04:34:07.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:07 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:34:07.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:07 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:34:07.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:07 smithi183 ceph-mon[33745]: osdmap e749: 8 total, 8 up, 8 in 2024-03-26T04:34:07.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:07 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:34:07.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:07 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-154","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-03-26T04:34:08.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:08 smithi016 ceph-mon[29101]: pgmap v1170: 268 pgs: 26 creating+peering, 6 unknown, 236 active+clean; 4.4 MiB data, 721 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-26T04:34:08.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:08 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:34:08.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:08 smithi016 ceph-mon[29101]: osdmap e750: 8 total, 8 up, 8 in 2024-03-26T04:34:08.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:08 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-111","var": "dedup_tier","val": "test-rados-api-smithi016-74534-154"}]: dispatch 2024-03-26T04:34:08.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:08 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-111","var": "dedup_tier","val": "test-rados-api-smithi016-74534-154"}]: dispatch 2024-03-26T04:34:08.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:08 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:34:08.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:08 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:34:08.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:08 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-154"}]: dispatch 2024-03-26T04:34:08.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:08 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-154"}]: dispatch 2024-03-26T04:34:08.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:08 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:34:08.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:08 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:34:08.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:08 smithi016 ceph-mon[37215]: pgmap v1170: 268 pgs: 26 creating+peering, 6 unknown, 236 active+clean; 4.4 MiB data, 721 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-26T04:34:08.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:08 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:34:08.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:08 smithi016 ceph-mon[37215]: osdmap e750: 8 total, 8 up, 8 in 2024-03-26T04:34:08.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:08 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-111","var": "dedup_tier","val": "test-rados-api-smithi016-74534-154"}]: dispatch 2024-03-26T04:34:08.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:08 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-111","var": "dedup_tier","val": "test-rados-api-smithi016-74534-154"}]: dispatch 2024-03-26T04:34:08.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:08 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:34:08.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:08 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:34:08.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:08 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-154"}]: dispatch 2024-03-26T04:34:08.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:08 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-154"}]: dispatch 2024-03-26T04:34:08.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:08 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:34:08.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:08 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:34:08.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:08 smithi183 ceph-mon[33745]: pgmap v1170: 268 pgs: 26 creating+peering, 6 unknown, 236 active+clean; 4.4 MiB data, 721 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-26T04:34:08.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:08 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi016-74534-154","app": "rados","yes_i_really_mean_it": true}]': finished 2024-03-26T04:34:08.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:08 smithi183 ceph-mon[33745]: osdmap e750: 8 total, 8 up, 8 in 2024-03-26T04:34:08.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:08 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-111","var": "dedup_tier","val": "test-rados-api-smithi016-74534-154"}]: dispatch 2024-03-26T04:34:08.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:08 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd pool set","pool":"test-rados-api-smithi016-74534-111","var": "dedup_tier","val": "test-rados-api-smithi016-74534-154"}]: dispatch 2024-03-26T04:34:08.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:08 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:34:08.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:08 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove-overlay", "pool": "test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:34:08.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:08 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-154"}]: dispatch 2024-03-26T04:34:08.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:08 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd tier remove", "pool": "test-rados-api-smithi016-74534-111", "tierpool": "test-rados-api-smithi016-74534-154"}]: dispatch 2024-03-26T04:34:08.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:08 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:34:08.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:08 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:34:09.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:34:08 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:34:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:34:09.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:09 smithi016 ceph-mon[29101]: osdmap e751: 8 total, 8 up, 8 in 2024-03-26T04:34:09.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:09 smithi016 ceph-mon[37215]: osdmap e751: 8 total, 8 up, 8 in 2024-03-26T04:34:09.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:09 smithi183 ceph-mon[33745]: osdmap e751: 8 total, 8 up, 8 in 2024-03-26T04:34:10.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:10 smithi016 ceph-mon[29101]: pgmap v1173: 236 pgs: 236 active+clean; 4.4 MiB data, 721 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-26T04:34:10.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:10 smithi016 ceph-mon[29101]: osdmap e752: 8 total, 8 up, 8 in 2024-03-26T04:34:10.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:10 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:34:10.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:10 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:34:10.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:10 smithi016 ceph-mon[37215]: pgmap v1173: 236 pgs: 236 active+clean; 4.4 MiB data, 721 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-26T04:34:10.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:10 smithi016 ceph-mon[37215]: osdmap e752: 8 total, 8 up, 8 in 2024-03-26T04:34:10.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:10 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:34:10.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:10 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:34:10.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:10 smithi183 ceph-mon[33745]: pgmap v1173: 236 pgs: 236 active+clean; 4.4 MiB data, 721 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-26T04:34:10.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:10 smithi183 ceph-mon[33745]: osdmap e752: 8 total, 8 up, 8 in 2024-03-26T04:34:10.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:10 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:34:10.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:10 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:34:11.362 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (7835 ms) 2024-03-26T04:34:11.362 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-03-26T04:34:11.362 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (13049 ms) 2024-03-26T04:34:11.362 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-03-26T04:34:11.362 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (8052 ms) 2024-03-26T04:34:11.362 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-03-26T04:34:11.362 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (13094 ms) 2024-03-26T04:34:11.362 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-03-26T04:34:11.362 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (7827 ms) 2024-03-26T04:34:11.362 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-03-26T04:34:11.363 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (8056 ms) 2024-03-26T04:34:11.363 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-03-26T04:34:11.363 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: hmm, no HitSet yet 2024-03-26T04:34:11.363 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: ok, hit_set contains 322:602f83fe:::foo:head 2024-03-26T04:34:11.363 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (9023 ms) 2024-03-26T04:34:11.363 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-03-26T04:34:11.363 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: got ls 1711427566,0 2024-03-26T04:34:11.363 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: first is 1711427566 2024-03-26T04:34:11.363 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: got ls 1711427566,0 2024-03-26T04:34:11.363 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: got ls 1711427566,0 2024-03-26T04:34:11.363 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: got ls 1711427566,1711427568,1711427569,0 2024-03-26T04:34:11.363 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: got ls 1711427566,1711427568,1711427569,0 2024-03-26T04:34:11.363 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: got ls 1711427566,1711427568,1711427569,0 2024-03-26T04:34:11.363 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: got ls 1711427566,1711427568,1711427569,1711427571,1711427572,0 2024-03-26T04:34:11.364 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: got ls 1711427566,1711427568,1711427569,1711427571,1711427572,0 2024-03-26T04:34:11.364 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: got ls 1711427566,1711427568,1711427569,1711427571,1711427572,0 2024-03-26T04:34:11.364 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: got ls 1711427566,1711427568,1711427569,1711427571,1711427572,1711427574,1711427575,0 2024-03-26T04:34:11.364 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: got ls 1711427566,1711427568,1711427569,1711427571,1711427572,1711427574,1711427575,0 2024-03-26T04:34:11.364 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: got ls 1711427566,1711427568,1711427569,1711427571,1711427572,1711427574,1711427575,0 2024-03-26T04:34:11.364 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: got ls 1711427569,1711427571,1711427572,1711427574,1711427575,1711427577,1711427578,0 2024-03-26T04:34:11.364 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: first now 1711427569, trimmed 2024-03-26T04:34:11.364 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (21066 ms) 2024-03-26T04:34:11.364 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-03-26T04:34:11.364 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: foo0 2024-03-26T04:34:11.364 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-03-26T04:34:11.364 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (14062 ms) 2024-03-26T04:34:11.364 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-03-26T04:34:11.364 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (17897 ms) 2024-03-26T04:34:11.365 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-03-26T04:34:11.365 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (22912 ms) 2024-03-26T04:34:11.365 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-03-26T04:34:11.365 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (5030 ms) 2024-03-26T04:34:11.365 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-03-26T04:34:11.365 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (3025 ms) 2024-03-26T04:34:11.365 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-03-26T04:34:11.365 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (3022 ms) 2024-03-26T04:34:11.365 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-03-26T04:34:11.365 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (3016 ms) 2024-03-26T04:34:11.365 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (232604 ms total) 2024-03-26T04:34:11.365 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: 2024-03-26T04:34:11.365 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-03-26T04:34:11.365 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (862714 ms total) 2024-03-26T04:34:11.365 INFO:tasks.workunit.client.0.smithi016.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-03-26T04:34:11.367 INFO:tasks.workunit.client.0.smithi016.stderr:+ for t in "${!pids[@]}" 2024-03-26T04:34:11.367 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74234 2024-03-26T04:34:11.367 INFO:tasks.workunit.client.0.smithi016.stderr:+ wait 74234 2024-03-26T04:34:11.367 INFO:tasks.workunit.client.0.smithi016.stderr:+ for t in "${!pids[@]}" 2024-03-26T04:34:11.367 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74439 2024-03-26T04:34:11.367 INFO:tasks.workunit.client.0.smithi016.stderr:+ wait 74439 2024-03-26T04:34:11.367 INFO:tasks.workunit.client.0.smithi016.stderr:+ for t in "${!pids[@]}" 2024-03-26T04:34:11.367 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=75016 2024-03-26T04:34:11.367 INFO:tasks.workunit.client.0.smithi016.stderr:+ wait 75016 2024-03-26T04:34:11.367 INFO:tasks.workunit.client.0.smithi016.stderr:+ for t in "${!pids[@]}" 2024-03-26T04:34:11.367 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74709 2024-03-26T04:34:11.367 INFO:tasks.workunit.client.0.smithi016.stderr:+ wait 74709 2024-03-26T04:34:11.368 INFO:tasks.workunit.client.0.smithi016.stderr:+ for t in "${!pids[@]}" 2024-03-26T04:34:11.368 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74575 2024-03-26T04:34:11.368 INFO:tasks.workunit.client.0.smithi016.stderr:+ wait 74575 2024-03-26T04:34:11.368 INFO:tasks.workunit.client.0.smithi016.stderr:+ for t in "${!pids[@]}" 2024-03-26T04:34:11.368 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74323 2024-03-26T04:34:11.368 INFO:tasks.workunit.client.0.smithi016.stderr:+ wait 74323 2024-03-26T04:34:11.368 INFO:tasks.workunit.client.0.smithi016.stderr:+ for t in "${!pids[@]}" 2024-03-26T04:34:11.368 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74630 2024-03-26T04:34:11.368 INFO:tasks.workunit.client.0.smithi016.stderr:+ wait 74630 2024-03-26T04:34:11.368 INFO:tasks.workunit.client.0.smithi016.stderr:+ for t in "${!pids[@]}" 2024-03-26T04:34:11.368 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=75275 2024-03-26T04:34:11.368 INFO:tasks.workunit.client.0.smithi016.stderr:+ wait 75275 2024-03-26T04:34:11.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:11 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:34:11.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:11 smithi016 ceph-mon[29101]: osdmap e753: 8 total, 8 up, 8 in 2024-03-26T04:34:11.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:11 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:34:11.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:11 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:34:11.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:11 smithi016 ceph-mon[29101]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:34:11.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:11 smithi016 ceph-mon[29101]: osdmap e754: 8 total, 8 up, 8 in 2024-03-26T04:34:11.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:11 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:34:11.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:11 smithi016 ceph-mon[37215]: osdmap e753: 8 total, 8 up, 8 in 2024-03-26T04:34:11.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:11 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:34:11.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:11 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:34:11.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:11 smithi016 ceph-mon[37215]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:34:11.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:11 smithi016 ceph-mon[37215]: osdmap e754: 8 total, 8 up, 8 in 2024-03-26T04:34:11.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:11 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:34:11.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:11 smithi183 ceph-mon[33745]: osdmap e753: 8 total, 8 up, 8 in 2024-03-26T04:34:11.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:11 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1201427292' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:34:11.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:11 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi016-74534-111"}]: dispatch 2024-03-26T04:34:11.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:11 smithi183 ceph-mon[33745]: from='client.56711 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"test-rados-api-smithi016-74534-111"}]': finished 2024-03-26T04:34:11.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:11 smithi183 ceph-mon[33745]: osdmap e754: 8 total, 8 up, 8 in 2024-03-26T04:34:12.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:12 smithi016 ceph-mon[37215]: pgmap v1176: 228 pgs: 228 active+clean; 582 KiB data, 721 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:34:12.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:12 smithi016 ceph-mon[37215]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:34:12.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:12 smithi016 ceph-mon[29101]: pgmap v1176: 228 pgs: 228 active+clean; 582 KiB data, 721 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:34:12.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:12 smithi016 ceph-mon[29101]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:34:12.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:12 smithi183 ceph-mon[33745]: pgmap v1176: 228 pgs: 228 active+clean; 582 KiB data, 721 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:34:12.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:12 smithi183 ceph-mon[33745]: Health check update: 3 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:34:13.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:34:12 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:34:14.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:14 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:34:14.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:14 smithi016 ceph-mon[37215]: pgmap v1178: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:34:14.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:14 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:34:14.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:14 smithi016 ceph-mon[29101]: pgmap v1178: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:34:14.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:14 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:34:14.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:14 smithi183 ceph-mon[33745]: pgmap v1178: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:34:16.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:16 smithi016 ceph-mon[37215]: pgmap v1179: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:34:16.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:16 smithi016 ceph-mon[29101]: pgmap v1179: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:34:16.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:16 smithi183 ceph-mon[33745]: pgmap v1179: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail 2024-03-26T04:34:18.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:18 smithi016 ceph-mon[37215]: pgmap v1180: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T04:34:18.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:18 smithi016 ceph-mon[29101]: pgmap v1180: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T04:34:18.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:18 smithi183 ceph-mon[33745]: pgmap v1180: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T04:34:19.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:34:18 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:34:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:34:20.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:20 smithi016 ceph-mon[29101]: pgmap v1181: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:34:20.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:20 smithi016 ceph-mon[37215]: pgmap v1181: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:34:20.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:20 smithi183 ceph-mon[33745]: pgmap v1181: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:34:22.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:22 smithi016 ceph-mon[29101]: pgmap v1182: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:34:22.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:22 smithi016 ceph-mon[37215]: pgmap v1182: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:34:22.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:22 smithi183 ceph-mon[33745]: pgmap v1182: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:34:23.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:34:22 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:34:23.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:23 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:34:23.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:23 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:34:23.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:23 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:34:24.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:24 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:34:24.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:24 smithi016 ceph-mon[29101]: pgmap v1183: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T04:34:24.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:24 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:34:24.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:24 smithi016 ceph-mon[37215]: pgmap v1183: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T04:34:24.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:24 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:34:24.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:24 smithi183 ceph-mon[33745]: pgmap v1183: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T04:34:26.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:26 smithi016 ceph-mon[37215]: pgmap v1184: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:34:26.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:26 smithi016 ceph-mon[29101]: pgmap v1184: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:34:26.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:26 smithi183 ceph-mon[33745]: pgmap v1184: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:34:28.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:28 smithi016 ceph-mon[29101]: pgmap v1185: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:34:28.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:28 smithi016 ceph-mon[37215]: pgmap v1185: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:34:28.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:28 smithi183 ceph-mon[33745]: pgmap v1185: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:34:29.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:34:28 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:34:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:34:30.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:30 smithi016 ceph-mon[29101]: pgmap v1186: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:34:30.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:30 smithi016 ceph-mon[37215]: pgmap v1186: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:34:30.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:30 smithi183 ceph-mon[33745]: pgmap v1186: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:34:32.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:32 smithi016 ceph-mon[29101]: pgmap v1187: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:34:32.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:32 smithi016 ceph-mon[37215]: pgmap v1187: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:34:32.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:32 smithi183 ceph-mon[33745]: pgmap v1187: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:34:33.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:34:32 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:34:34.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:34 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:34:34.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:34 smithi016 ceph-mon[29101]: pgmap v1188: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:34:34.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:34 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:34:34.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:34 smithi016 ceph-mon[37215]: pgmap v1188: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:34:34.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:34 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:34:34.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:34 smithi183 ceph-mon[33745]: pgmap v1188: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:34:36.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:36 smithi016 ceph-mon[29101]: pgmap v1189: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:34:36.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:36 smithi016 ceph-mon[37215]: pgmap v1189: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:34:36.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:36 smithi183 ceph-mon[33745]: pgmap v1189: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:34:38.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:38 smithi016 ceph-mon[29101]: pgmap v1190: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:34:38.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:38 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:34:38.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:38 smithi016 ceph-mon[37215]: pgmap v1190: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:34:38.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:38 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:34:38.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:38 smithi183 ceph-mon[33745]: pgmap v1190: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:34:38.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:38 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:34:39.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:34:38 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:34:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:34:40.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:40 smithi016 ceph-mon[37215]: pgmap v1191: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:34:40.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:40 smithi016 ceph-mon[29101]: pgmap v1191: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:34:40.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:40 smithi183 ceph-mon[33745]: pgmap v1191: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:34:42.664 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:42 smithi016 ceph-mon[37215]: pgmap v1192: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:34:42.665 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:42 smithi016 ceph-mon[29101]: pgmap v1192: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:34:42.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:42 smithi183 ceph-mon[33745]: pgmap v1192: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:34:43.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:34:42 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:34:44.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:44 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:34:44.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:44 smithi016 ceph-mon[29101]: pgmap v1193: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:34:44.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:44 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:34:44.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:44 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:34:44.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:44 smithi016 ceph-mon[37215]: pgmap v1193: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:34:44.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:44 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:34:44.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:44 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:34:44.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:44 smithi183 ceph-mon[33745]: pgmap v1193: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:34:44.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:44 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:34:45.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:45 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:34:45.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:45 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:34:45.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:45 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:34:45.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:45 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:34:45.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:45 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:34:45.663 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:45 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:34:45.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:45 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:34:45.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:45 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:34:45.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:45 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:34:46.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:46 smithi016 ceph-mon[37215]: pgmap v1194: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:34:46.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:46 smithi016 ceph-mon[29101]: pgmap v1194: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:34:46.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:46 smithi183 ceph-mon[33745]: pgmap v1194: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:34:48.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:48 smithi016 ceph-mon[37215]: pgmap v1195: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:34:48.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:48 smithi016 ceph-mon[29101]: pgmap v1195: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:34:48.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:48 smithi183 ceph-mon[33745]: pgmap v1195: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:34:49.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:34:48 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:34:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:34:50.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:50 smithi016 ceph-mon[29101]: pgmap v1196: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:34:50.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:50 smithi016 ceph-mon[37215]: pgmap v1196: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:34:50.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:50 smithi183 ceph-mon[33745]: pgmap v1196: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:34:52.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:52 smithi183 ceph-mon[33745]: pgmap v1197: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:34:52.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:52 smithi016 ceph-mon[29101]: pgmap v1197: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:34:52.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:52 smithi016 ceph-mon[37215]: pgmap v1197: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:34:53.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:34:52 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:34:53.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:53 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:34:53.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:53 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:34:53.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:53 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:34:54.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:54 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:34:54.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:54 smithi183 ceph-mon[33745]: pgmap v1198: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:34:54.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:54 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:34:54.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:54 smithi016 ceph-mon[29101]: pgmap v1198: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:34:54.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:54 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:34:54.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:54 smithi016 ceph-mon[37215]: pgmap v1198: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:34:56.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:56 smithi183 ceph-mon[33745]: pgmap v1199: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:34:56.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:56 smithi016 ceph-mon[37215]: pgmap v1199: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:34:56.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:56 smithi016 ceph-mon[29101]: pgmap v1199: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:34:58.745 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:34:58 smithi016 ceph-mon[37215]: pgmap v1200: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:34:58.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:34:58 smithi016 ceph-mon[29101]: pgmap v1200: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:34:58.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:34:58 smithi183 ceph-mon[33745]: pgmap v1200: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:34:59.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:34:58 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:34:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:35:00.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:35:00 smithi183 ceph-mon[33745]: pgmap v1201: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:35:00.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:35:00 smithi016 ceph-mon[29101]: pgmap v1201: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:35:00.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:35:00 smithi016 ceph-mon[37215]: pgmap v1201: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:35:02.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:35:02 smithi183 ceph-mon[33745]: pgmap v1202: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:35:02.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:35:02 smithi016 ceph-mon[29101]: pgmap v1202: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:35:02.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:35:02 smithi016 ceph-mon[37215]: pgmap v1202: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:35:03.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:35:02 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:35:04.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:35:04 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:35:04.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:35:04 smithi183 ceph-mon[33745]: pgmap v1203: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:35:04.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:35:04 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:35:04.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:35:04 smithi016 ceph-mon[29101]: pgmap v1203: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:35:04.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:35:04 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:35:04.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:35:04 smithi016 ceph-mon[37215]: pgmap v1203: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:35:06.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:35:06 smithi183 ceph-mon[33745]: pgmap v1204: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:35:06.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:35:06 smithi016 ceph-mon[29101]: pgmap v1204: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:35:06.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:35:06 smithi016 ceph-mon[37215]: pgmap v1204: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:35:08.745 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:35:08 smithi016 ceph-mon[29101]: pgmap v1205: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:35:08.745 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:35:08 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:35:08.745 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:35:08 smithi016 ceph-mon[37215]: pgmap v1205: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:35:08.746 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:35:08 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:35:08.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:35:08 smithi183 ceph-mon[33745]: pgmap v1205: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:35:08.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:35:08 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:35:09.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:35:08 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:35:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:35:10.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:35:10 smithi183 ceph-mon[33745]: pgmap v1206: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:35:10.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:35:10 smithi016 ceph-mon[37215]: pgmap v1206: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:35:10.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:35:10 smithi016 ceph-mon[29101]: pgmap v1206: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:35:12.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:35:12 smithi183 ceph-mon[33745]: pgmap v1207: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:35:12.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:35:12 smithi016 ceph-mon[37215]: pgmap v1207: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:35:12.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:35:12 smithi016 ceph-mon[29101]: pgmap v1207: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:35:13.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:35:12 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:35:14.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:35:14 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:35:14.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:35:14 smithi183 ceph-mon[33745]: pgmap v1208: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:35:14.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:35:14 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:35:14.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:35:14 smithi016 ceph-mon[29101]: pgmap v1208: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:35:14.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:35:14 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:35:14.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:35:14 smithi016 ceph-mon[37215]: pgmap v1208: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:35:16.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:35:16 smithi183 ceph-mon[33745]: pgmap v1209: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:35:16.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:35:16 smithi016 ceph-mon[37215]: pgmap v1209: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:35:16.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:35:16 smithi016 ceph-mon[29101]: pgmap v1209: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:35:18.745 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:35:18 smithi016 ceph-mon[29101]: pgmap v1210: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:35:18.746 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:35:18 smithi016 ceph-mon[37215]: pgmap v1210: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:35:18.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:35:18 smithi183 ceph-mon[33745]: pgmap v1210: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:35:19.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:35:18 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:35:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:35:20.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:35:20 smithi183 ceph-mon[33745]: pgmap v1211: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:35:20.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:35:20 smithi016 ceph-mon[37215]: pgmap v1211: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:35:20.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:35:20 smithi016 ceph-mon[29101]: pgmap v1211: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:35:22.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:35:22 smithi183 ceph-mon[33745]: pgmap v1212: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:35:22.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:35:22 smithi016 ceph-mon[37215]: pgmap v1212: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:35:22.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:35:22 smithi016 ceph-mon[29101]: pgmap v1212: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:35:23.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:35:22 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:35:23.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:35:23 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:35:23.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:35:23 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:35:23.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:35:23 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:35:24.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:35:24 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:35:24.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:35:24 smithi183 ceph-mon[33745]: pgmap v1213: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:35:24.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:35:24 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:35:24.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:35:24 smithi016 ceph-mon[37215]: pgmap v1213: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:35:24.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:35:24 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:35:24.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:35:24 smithi016 ceph-mon[29101]: pgmap v1213: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:35:26.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:35:26 smithi183 ceph-mon[33745]: pgmap v1214: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:35:26.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:35:26 smithi016 ceph-mon[37215]: pgmap v1214: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:35:26.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:35:26 smithi016 ceph-mon[29101]: pgmap v1214: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:35:28.745 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:35:28 smithi016 ceph-mon[29101]: pgmap v1215: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:35:28.745 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:35:28 smithi016 ceph-mon[37215]: pgmap v1215: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:35:28.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:35:28 smithi183 ceph-mon[33745]: pgmap v1215: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:35:29.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:35:28 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:35:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:35:30.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:35:30 smithi183 ceph-mon[33745]: pgmap v1216: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:35:30.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:35:30 smithi016 ceph-mon[37215]: pgmap v1216: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:35:30.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:35:30 smithi016 ceph-mon[29101]: pgmap v1216: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:35:32.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:35:32 smithi183 ceph-mon[33745]: pgmap v1217: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:35:32.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:35:32 smithi016 ceph-mon[37215]: pgmap v1217: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:35:32.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:35:32 smithi016 ceph-mon[29101]: pgmap v1217: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:35:33.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:35:32 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:35:34.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:35:34 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:35:34.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:35:34 smithi183 ceph-mon[33745]: pgmap v1218: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:35:34.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:35:34 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:35:34.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:35:34 smithi016 ceph-mon[37215]: pgmap v1218: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:35:34.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:35:34 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:35:34.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:35:34 smithi016 ceph-mon[29101]: pgmap v1218: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:35:36.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:35:36 smithi183 ceph-mon[33745]: pgmap v1219: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:35:36.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:35:36 smithi016 ceph-mon[37215]: pgmap v1219: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:35:36.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:35:36 smithi016 ceph-mon[29101]: pgmap v1219: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:35:38.745 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:35:38 smithi016 ceph-mon[37215]: pgmap v1220: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:35:38.746 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:35:38 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:35:38.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:35:38 smithi016 ceph-mon[29101]: pgmap v1220: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:35:38.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:35:38 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:35:38.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:35:38 smithi183 ceph-mon[33745]: pgmap v1220: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:35:38.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:35:38 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:35:39.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:35:38 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:35:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:35:40.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:35:40 smithi183 ceph-mon[33745]: pgmap v1221: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:35:40.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:35:40 smithi016 ceph-mon[29101]: pgmap v1221: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:35:40.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:35:40 smithi016 ceph-mon[37215]: pgmap v1221: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:35:42.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:35:42 smithi183 ceph-mon[33745]: pgmap v1222: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:35:42.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:35:42 smithi016 ceph-mon[37215]: pgmap v1222: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:35:42.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:35:42 smithi016 ceph-mon[29101]: pgmap v1222: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:35:43.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:35:42 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:35:44.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:35:44 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:35:44.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:35:44 smithi183 ceph-mon[33745]: pgmap v1223: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:35:44.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:35:44 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:35:44.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:35:44 smithi016 ceph-mon[37215]: pgmap v1223: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:35:44.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:35:44 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:35:44.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:35:44 smithi016 ceph-mon[29101]: pgmap v1223: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:35:45.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:35:45 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:35:45.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:35:45 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:35:45.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:35:45 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:35:46.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:35:46 smithi183 ceph-mon[33745]: pgmap v1224: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:35:46.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:35:46 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:35:46.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:35:46 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:35:46.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:35:46 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:35:46.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:35:46 smithi016 ceph-mon[29101]: pgmap v1224: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:35:46.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:35:46 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:35:46.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:35:46 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:35:46.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:35:46 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:35:46.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:35:46 smithi016 ceph-mon[37215]: pgmap v1224: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:35:46.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:35:46 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:35:46.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:35:46 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:35:46.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:35:46 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:35:48.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:35:48 smithi183 ceph-mon[33745]: pgmap v1225: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:35:48.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:35:48 smithi016 ceph-mon[29101]: pgmap v1225: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:35:48.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:35:48 smithi016 ceph-mon[37215]: pgmap v1225: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:35:48.913 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:35:48 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:35:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:35:50.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:35:50 smithi183 ceph-mon[33745]: pgmap v1226: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:35:50.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:35:50 smithi016 ceph-mon[29101]: pgmap v1226: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:35:50.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:35:50 smithi016 ceph-mon[37215]: pgmap v1226: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:35:52.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:35:52 smithi183 ceph-mon[33745]: pgmap v1227: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:35:52.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:35:52 smithi016 ceph-mon[29101]: pgmap v1227: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:35:52.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:35:52 smithi016 ceph-mon[37215]: pgmap v1227: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:35:53.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:35:52 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:35:53.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:35:53 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:35:53.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:35:53 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:35:53.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:35:53 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:35:54.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:35:54 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:35:54.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:35:54 smithi183 ceph-mon[33745]: pgmap v1228: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:35:54.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:35:54 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:35:54.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:35:54 smithi016 ceph-mon[29101]: pgmap v1228: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:35:54.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:35:54 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:35:54.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:35:54 smithi016 ceph-mon[37215]: pgmap v1228: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:35:56.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:35:56 smithi183 ceph-mon[33745]: pgmap v1229: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:35:56.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:35:56 smithi016 ceph-mon[29101]: pgmap v1229: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:35:56.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:35:56 smithi016 ceph-mon[37215]: pgmap v1229: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:35:58.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:35:58 smithi183 ceph-mon[33745]: pgmap v1230: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:35:58.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:35:58 smithi016 ceph-mon[29101]: pgmap v1230: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:35:58.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:35:58 smithi016 ceph-mon[37215]: pgmap v1230: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:35:58.913 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:35:58 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:35:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:36:00.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:36:00 smithi183 ceph-mon[33745]: pgmap v1231: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:36:00.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:36:00 smithi016 ceph-mon[29101]: pgmap v1231: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:36:00.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:36:00 smithi016 ceph-mon[37215]: pgmap v1231: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:36:02.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:36:02 smithi183 ceph-mon[33745]: pgmap v1232: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:36:02.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:36:02 smithi016 ceph-mon[29101]: pgmap v1232: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:36:02.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:36:02 smithi016 ceph-mon[37215]: pgmap v1232: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:36:03.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:36:02 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:36:04.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:36:04 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:36:04.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:36:04 smithi183 ceph-mon[33745]: pgmap v1233: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:36:04.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:36:04 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:36:04.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:36:04 smithi016 ceph-mon[37215]: pgmap v1233: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:36:04.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:36:04 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:36:04.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:36:04 smithi016 ceph-mon[29101]: pgmap v1233: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:36:06.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:36:06 smithi183 ceph-mon[33745]: pgmap v1234: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:36:06.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:36:06 smithi016 ceph-mon[29101]: pgmap v1234: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:36:06.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:36:06 smithi016 ceph-mon[37215]: pgmap v1234: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:36:08.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:36:08 smithi183 ceph-mon[33745]: pgmap v1235: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:36:08.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:36:08 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:36:08.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:36:08 smithi016 ceph-mon[37215]: pgmap v1235: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:36:08.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:36:08 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:36:08.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:36:08 smithi016 ceph-mon[29101]: pgmap v1235: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:36:08.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:36:08 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:36:08.913 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:36:08 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:36:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:36:10.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:36:10 smithi183 ceph-mon[33745]: pgmap v1236: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:36:10.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:36:10 smithi016 ceph-mon[37215]: pgmap v1236: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:36:10.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:36:10 smithi016 ceph-mon[29101]: pgmap v1236: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:36:12.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:36:12 smithi183 ceph-mon[33745]: pgmap v1237: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:36:12.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:36:12 smithi016 ceph-mon[29101]: pgmap v1237: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:36:12.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:36:12 smithi016 ceph-mon[37215]: pgmap v1237: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:36:13.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:36:12 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:36:14.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:36:14 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:36:14.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:36:14 smithi016 ceph-mon[29101]: pgmap v1238: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:36:14.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:36:14 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:36:14.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:36:14 smithi016 ceph-mon[37215]: pgmap v1238: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:36:15.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:36:14 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:36:15.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:36:14 smithi183 ceph-mon[33745]: pgmap v1238: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:36:16.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:36:16 smithi016 ceph-mon[29101]: pgmap v1239: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:36:16.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:36:16 smithi016 ceph-mon[37215]: pgmap v1239: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:36:17.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:36:16 smithi183 ceph-mon[33745]: pgmap v1239: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:36:18.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:36:18 smithi016 ceph-mon[29101]: pgmap v1240: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:36:18.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:36:18 smithi016 ceph-mon[37215]: pgmap v1240: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:36:18.913 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:36:18 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:36:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:36:19.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:36:18 smithi183 ceph-mon[33745]: pgmap v1240: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:36:20.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:36:20 smithi016 ceph-mon[29101]: pgmap v1241: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:36:20.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:36:20 smithi016 ceph-mon[37215]: pgmap v1241: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:36:21.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:36:20 smithi183 ceph-mon[33745]: pgmap v1241: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:36:22.900 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:36:22 smithi183 ceph-mon[33745]: pgmap v1242: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:36:22.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:36:22 smithi016 ceph-mon[37215]: pgmap v1242: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:36:22.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:36:22 smithi016 ceph-mon[29101]: pgmap v1242: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:36:23.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:36:22 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:36:23.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:36:23 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:36:23.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:36:23 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:36:24.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:36:23 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:36:24.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:36:24 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:36:24.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:36:24 smithi016 ceph-mon[29101]: pgmap v1243: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:36:24.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:36:24 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:36:24.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:36:24 smithi016 ceph-mon[37215]: pgmap v1243: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:36:25.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:36:24 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:36:25.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:36:24 smithi183 ceph-mon[33745]: pgmap v1243: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:36:26.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:36:26 smithi016 ceph-mon[29101]: pgmap v1244: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:36:26.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:36:26 smithi016 ceph-mon[37215]: pgmap v1244: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:36:27.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:36:26 smithi183 ceph-mon[33745]: pgmap v1244: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:36:28.843 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:36:28 smithi183 ceph-mon[33745]: pgmap v1245: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:36:28.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:36:28 smithi016 ceph-mon[29101]: pgmap v1245: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:36:28.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:36:28 smithi016 ceph-mon[37215]: pgmap v1245: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:36:28.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:36:28 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:36:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:36:30.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:36:30 smithi016 ceph-mon[29101]: pgmap v1246: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:36:30.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:36:30 smithi016 ceph-mon[37215]: pgmap v1246: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:36:31.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:36:30 smithi183 ceph-mon[33745]: pgmap v1246: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:36:32.910 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:36:32 smithi183 ceph-mon[33745]: pgmap v1247: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:36:32.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:36:32 smithi016 ceph-mon[29101]: pgmap v1247: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:36:32.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:36:32 smithi016 ceph-mon[37215]: pgmap v1247: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:36:33.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:36:32 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:36:34.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:36:34 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:36:34.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:36:34 smithi016 ceph-mon[37215]: pgmap v1248: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:36:34.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:36:34 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:36:34.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:36:34 smithi016 ceph-mon[29101]: pgmap v1248: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:36:35.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:36:34 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:36:35.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:36:34 smithi183 ceph-mon[33745]: pgmap v1248: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:36:36.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:36:36 smithi016 ceph-mon[37215]: pgmap v1249: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:36:36.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:36:36 smithi016 ceph-mon[29101]: pgmap v1249: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:36:37.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:36:36 smithi183 ceph-mon[33745]: pgmap v1249: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:36:38.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:36:38 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:36:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:36:38.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:36:38 smithi016 ceph-mon[29101]: pgmap v1250: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:36:38.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:36:38 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:36:38.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:36:38 smithi016 ceph-mon[37215]: pgmap v1250: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:36:38.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:36:38 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:36:39.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:36:38 smithi183 ceph-mon[33745]: pgmap v1250: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:36:39.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:36:38 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:36:40.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:36:40 smithi016 ceph-mon[29101]: pgmap v1251: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:36:40.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:36:40 smithi016 ceph-mon[37215]: pgmap v1251: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:36:41.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:36:40 smithi183 ceph-mon[33745]: pgmap v1251: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:36:42.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:36:42 smithi016 ceph-mon[29101]: pgmap v1252: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:36:42.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:36:42 smithi016 ceph-mon[37215]: pgmap v1252: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:36:42.919 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:36:42 smithi183 ceph-mon[33745]: pgmap v1252: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:36:43.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:36:42 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:36:44.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:36:44 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:36:44.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:36:44 smithi016 ceph-mon[29101]: pgmap v1253: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:36:44.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:36:44 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:36:44.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:36:44 smithi016 ceph-mon[37215]: pgmap v1253: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:36:45.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:36:44 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:36:45.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:36:44 smithi183 ceph-mon[33745]: pgmap v1253: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:36:46.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:36:46 smithi016 ceph-mon[29101]: pgmap v1254: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:36:46.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:36:46 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:36:46.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:36:46 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:36:46.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:36:46 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:36:46.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:36:46 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:36:46.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:36:46 smithi016 ceph-mon[37215]: pgmap v1254: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:36:46.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:36:46 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:36:46.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:36:46 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:36:46.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:36:46 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:36:46.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:36:46 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:36:47.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:36:46 smithi183 ceph-mon[33745]: pgmap v1254: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:36:47.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:36:46 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:36:47.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:36:46 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:36:47.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:36:46 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:36:47.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:36:46 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:36:48.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:36:48 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:36:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:36:48.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:36:48 smithi016 ceph-mon[37215]: pgmap v1255: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:36:48.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:36:48 smithi016 ceph-mon[29101]: pgmap v1255: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:36:49.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:36:48 smithi183 ceph-mon[33745]: pgmap v1255: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:36:50.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:36:50 smithi016 ceph-mon[37215]: pgmap v1256: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:36:50.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:36:50 smithi016 ceph-mon[29101]: pgmap v1256: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:36:51.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:36:50 smithi183 ceph-mon[33745]: pgmap v1256: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:36:52.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:36:52 smithi016 ceph-mon[37215]: pgmap v1257: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:36:52.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:36:52 smithi016 ceph-mon[29101]: pgmap v1257: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:36:52.924 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:36:52 smithi183 ceph-mon[33745]: pgmap v1257: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:36:53.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:36:52 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:36:53.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:36:53 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:36:53.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:36:53 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:36:54.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:36:53 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:36:54.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:36:54 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:36:54.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:36:54 smithi016 ceph-mon[37215]: pgmap v1258: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:36:54.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:36:54 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:36:54.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:36:54 smithi016 ceph-mon[29101]: pgmap v1258: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:36:55.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:36:54 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:36:55.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:36:54 smithi183 ceph-mon[33745]: pgmap v1258: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:36:56.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:36:56 smithi016 ceph-mon[37215]: pgmap v1259: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:36:56.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:36:56 smithi016 ceph-mon[29101]: pgmap v1259: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:36:57.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:36:56 smithi183 ceph-mon[33745]: pgmap v1259: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:36:58.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:36:58 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:36:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:36:58.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:36:58 smithi016 ceph-mon[37215]: pgmap v1260: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:36:58.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:36:58 smithi016 ceph-mon[29101]: pgmap v1260: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:36:59.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:36:58 smithi183 ceph-mon[33745]: pgmap v1260: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:00.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:37:00 smithi016 ceph-mon[29101]: pgmap v1261: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:37:00.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:37:00 smithi016 ceph-mon[37215]: pgmap v1261: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:37:01.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:37:00 smithi183 ceph-mon[33745]: pgmap v1261: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:37:02.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:37:02 smithi016 ceph-mon[29101]: pgmap v1262: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:37:02.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:37:02 smithi016 ceph-mon[37215]: pgmap v1262: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:37:02.935 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:37:02 smithi183 ceph-mon[33745]: pgmap v1262: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:37:03.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:37:02 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:37:04.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:37:04 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:37:04.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:37:04 smithi016 ceph-mon[37215]: pgmap v1263: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:04.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:37:04 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:37:04.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:37:04 smithi016 ceph-mon[29101]: pgmap v1263: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:05.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:37:04 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:37:05.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:37:04 smithi183 ceph-mon[33745]: pgmap v1263: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:06.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:37:06 smithi016 ceph-mon[37215]: pgmap v1264: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:37:06.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:37:06 smithi016 ceph-mon[29101]: pgmap v1264: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:37:07.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:37:06 smithi183 ceph-mon[33745]: pgmap v1264: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:37:08.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:37:08 smithi016 ceph-mon[37215]: pgmap v1265: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:08.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:37:08 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:37:08.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:37:08 smithi016 ceph-mon[29101]: pgmap v1265: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:08.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:37:08 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:37:08.913 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:37:08 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:37:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:37:09.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:37:08 smithi183 ceph-mon[33745]: pgmap v1265: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:09.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:37:08 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:37:10.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:37:10 smithi016 ceph-mon[37215]: pgmap v1266: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:37:10.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:37:10 smithi016 ceph-mon[29101]: pgmap v1266: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:37:11.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:37:10 smithi183 ceph-mon[33745]: pgmap v1266: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:37:12.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:37:12 smithi016 ceph-mon[29101]: pgmap v1267: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:37:12.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:37:12 smithi016 ceph-mon[37215]: pgmap v1267: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:37:12.947 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:37:12 smithi183 ceph-mon[33745]: pgmap v1267: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:37:13.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:37:12 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:37:14.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:37:14 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:37:14.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:37:14 smithi016 ceph-mon[37215]: pgmap v1268: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:14.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:37:14 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:37:14.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:37:14 smithi016 ceph-mon[29101]: pgmap v1268: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:15.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:37:14 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:37:15.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:37:14 smithi183 ceph-mon[33745]: pgmap v1268: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:16.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:37:16 smithi016 ceph-mon[37215]: pgmap v1269: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:37:16.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:37:16 smithi016 ceph-mon[29101]: pgmap v1269: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:37:17.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:37:16 smithi183 ceph-mon[33745]: pgmap v1269: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:37:18.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:37:18 smithi016 ceph-mon[29101]: pgmap v1270: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:18.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:37:18 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:37:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:37:18.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:37:18 smithi016 ceph-mon[37215]: pgmap v1270: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:19.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:37:18 smithi183 ceph-mon[33745]: pgmap v1270: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:20.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:37:20 smithi016 ceph-mon[37215]: pgmap v1271: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:37:20.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:37:20 smithi016 ceph-mon[29101]: pgmap v1271: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:37:21.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:37:20 smithi183 ceph-mon[33745]: pgmap v1271: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:37:22.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:37:22 smithi016 ceph-mon[37215]: pgmap v1272: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:37:22.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:37:22 smithi016 ceph-mon[29101]: pgmap v1272: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:37:22.958 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:37:22 smithi183 ceph-mon[33745]: pgmap v1272: 228 pgs: 228 active+clean; 582 KiB data, 733 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:37:23.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:37:22 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:37:23.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:37:23 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:37:23.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:37:23 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:37:24.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:37:23 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:37:24.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:37:24 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:37:24.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:37:24 smithi016 ceph-mon[37215]: pgmap v1273: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T04:37:24.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:37:24 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:37:24.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:37:24 smithi016 ceph-mon[29101]: pgmap v1273: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T04:37:25.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:37:24 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:37:25.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:37:24 smithi183 ceph-mon[33745]: pgmap v1273: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T04:37:26.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:37:26 smithi016 ceph-mon[29101]: pgmap v1274: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T04:37:26.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:37:26 smithi016 ceph-mon[37215]: pgmap v1274: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T04:37:27.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:37:26 smithi183 ceph-mon[33745]: pgmap v1274: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T04:37:28.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:37:28 smithi016 ceph-mon[37215]: pgmap v1275: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T04:37:28.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:37:28 smithi016 ceph-mon[29101]: pgmap v1275: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T04:37:28.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:37:28 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:37:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:37:29.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:37:28 smithi183 ceph-mon[33745]: pgmap v1275: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T04:37:30.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:37:30 smithi016 ceph-mon[29101]: pgmap v1276: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T04:37:30.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:37:30 smithi016 ceph-mon[37215]: pgmap v1276: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T04:37:31.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:37:30 smithi183 ceph-mon[33745]: pgmap v1276: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T04:37:32.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:37:32 smithi016 ceph-mon[37215]: pgmap v1277: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T04:37:32.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:37:32 smithi016 ceph-mon[29101]: pgmap v1277: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T04:37:32.969 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:37:32 smithi183 ceph-mon[33745]: pgmap v1277: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T04:37:33.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:37:32 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:37:34.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:37:34 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:37:34.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:37:34 smithi016 ceph-mon[37215]: pgmap v1278: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T04:37:34.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:37:34 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:37:34.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:37:34 smithi016 ceph-mon[29101]: pgmap v1278: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T04:37:35.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:37:34 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:37:35.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:37:34 smithi183 ceph-mon[33745]: pgmap v1278: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T04:37:37.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:37:36 smithi183 ceph-mon[33745]: pgmap v1279: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:37:37.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:37:36 smithi016 ceph-mon[37215]: pgmap v1279: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:37:37.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:37:36 smithi016 ceph-mon[29101]: pgmap v1279: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:37:39.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:37:38 smithi183 ceph-mon[33745]: pgmap v1280: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:39.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:37:38 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:37:39.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:37:38 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:37:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:37:39.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:37:38 smithi016 ceph-mon[29101]: pgmap v1280: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:39.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:37:38 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:37:39.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:37:38 smithi016 ceph-mon[37215]: pgmap v1280: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:39.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:37:38 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:37:41.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:37:40 smithi183 ceph-mon[33745]: pgmap v1281: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:37:41.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:37:40 smithi016 ceph-mon[37215]: pgmap v1281: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:37:41.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:37:40 smithi016 ceph-mon[29101]: pgmap v1281: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:37:42.979 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:37:42 smithi183 ceph-mon[33745]: pgmap v1282: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:37:43.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:37:42 smithi016 ceph-mon[37215]: pgmap v1282: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:37:43.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:37:42 smithi016 ceph-mon[29101]: pgmap v1282: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:37:43.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:37:42 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:37:45.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:37:44 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:37:45.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:37:44 smithi183 ceph-mon[33745]: pgmap v1283: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:45.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:37:44 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:37:45.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:37:44 smithi016 ceph-mon[37215]: pgmap v1283: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:45.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:37:44 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:37:45.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:37:44 smithi016 ceph-mon[29101]: pgmap v1283: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:47.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:37:46 smithi183 ceph-mon[33745]: pgmap v1284: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:37:47.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:37:46 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:37:47.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:37:46 smithi016 ceph-mon[29101]: pgmap v1284: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:37:47.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:37:46 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:37:47.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:37:46 smithi016 ceph-mon[37215]: pgmap v1284: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:37:47.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:37:46 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:37:48.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:37:47 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:37:48.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:37:47 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:37:48.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:37:47 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:37:48.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:37:47 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:37:48.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:37:47 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:37:48.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:37:47 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:37:48.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:37:47 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:37:48.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:37:47 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:37:48.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:37:47 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:37:49.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:37:48 smithi183 ceph-mon[33745]: pgmap v1285: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:49.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:37:48 smithi016 ceph-mon[29101]: pgmap v1285: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:49.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:37:48 smithi016 ceph-mon[37215]: pgmap v1285: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:49.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:37:48 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:37:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:37:51.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:37:50 smithi183 ceph-mon[33745]: pgmap v1286: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:37:51.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:37:50 smithi016 ceph-mon[29101]: pgmap v1286: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:37:51.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:37:50 smithi016 ceph-mon[37215]: pgmap v1286: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:37:52.990 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:37:52 smithi183 ceph-mon[33745]: pgmap v1287: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:37:53.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:37:52 smithi016 ceph-mon[37215]: pgmap v1287: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:37:53.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:37:52 smithi016 ceph-mon[29101]: pgmap v1287: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:37:53.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:37:52 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:37:54.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:37:53 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:37:54.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:37:53 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:37:54.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:37:53 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:37:55.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:37:54 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:37:55.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:37:54 smithi183 ceph-mon[33745]: pgmap v1288: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:55.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:37:54 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:37:55.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:37:54 smithi016 ceph-mon[37215]: pgmap v1288: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:55.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:37:54 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:37:55.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:37:54 smithi016 ceph-mon[29101]: pgmap v1288: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:57.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:37:56 smithi183 ceph-mon[33745]: pgmap v1289: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:37:57.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:37:56 smithi016 ceph-mon[37215]: pgmap v1289: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:37:57.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:37:56 smithi016 ceph-mon[29101]: pgmap v1289: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:37:59.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:37:58 smithi183 ceph-mon[33745]: pgmap v1290: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:59.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:37:58 smithi016 ceph-mon[29101]: pgmap v1290: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:37:59.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:37:58 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:37:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:37:59.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:37:58 smithi016 ceph-mon[37215]: pgmap v1290: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:00.964 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:38:00 smithi016 ceph-mon[37215]: pgmap v1291: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:00.964 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:38:00 smithi016 ceph-mon[29101]: pgmap v1291: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:01.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:38:00 smithi183 ceph-mon[33745]: pgmap v1291: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:03.001 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:38:02 smithi183 ceph-mon[33745]: pgmap v1292: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:03.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:38:02 smithi016 ceph-mon[29101]: pgmap v1292: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:03.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:38:02 smithi016 ceph-mon[37215]: pgmap v1292: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:03.293 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:38:03 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:38:05.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:38:04 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:38:05.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:38:04 smithi183 ceph-mon[33745]: pgmap v1293: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:05.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:38:04 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:38:05.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:38:04 smithi016 ceph-mon[37215]: pgmap v1293: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:05.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:38:04 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:38:05.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:38:04 smithi016 ceph-mon[29101]: pgmap v1293: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:07.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:38:06 smithi183 ceph-mon[33745]: pgmap v1294: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:07.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:38:06 smithi016 ceph-mon[37215]: pgmap v1294: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:07.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:38:06 smithi016 ceph-mon[29101]: pgmap v1294: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:09.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:38:08 smithi183 ceph-mon[33745]: pgmap v1295: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:09.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:38:08 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:38:09.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:38:08 smithi016 ceph-mon[29101]: pgmap v1295: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:09.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:38:08 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:38:09.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:38:08 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:38:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:38:09.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:38:08 smithi016 ceph-mon[37215]: pgmap v1295: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:09.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:38:08 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:38:11.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:38:10 smithi183 ceph-mon[33745]: pgmap v1296: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:11.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:38:10 smithi016 ceph-mon[37215]: pgmap v1296: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:11.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:38:10 smithi016 ceph-mon[29101]: pgmap v1296: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:13.012 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:38:12 smithi183 ceph-mon[33745]: pgmap v1297: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:13.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:38:12 smithi016 ceph-mon[29101]: pgmap v1297: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:13.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:38:12 smithi016 ceph-mon[37215]: pgmap v1297: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:13.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:38:13 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:38:15.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:38:14 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:38:15.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:38:14 smithi183 ceph-mon[33745]: pgmap v1298: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:15.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:38:14 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:38:15.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:38:14 smithi016 ceph-mon[29101]: pgmap v1298: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:15.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:38:14 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:38:15.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:38:14 smithi016 ceph-mon[37215]: pgmap v1298: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:17.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:38:16 smithi183 ceph-mon[33745]: pgmap v1299: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:38:17.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:38:16 smithi016 ceph-mon[29101]: pgmap v1299: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:38:17.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:38:16 smithi016 ceph-mon[37215]: pgmap v1299: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:38:19.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:38:18 smithi183 ceph-mon[33745]: pgmap v1300: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:19.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:38:18 smithi016 ceph-mon[29101]: pgmap v1300: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:19.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:38:18 smithi016 ceph-mon[37215]: pgmap v1300: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:19.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:38:18 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:38:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:38:21.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:38:20 smithi183 ceph-mon[33745]: pgmap v1301: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:21.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:38:20 smithi016 ceph-mon[37215]: pgmap v1301: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:21.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:38:20 smithi016 ceph-mon[29101]: pgmap v1301: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:23.023 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:38:22 smithi183 ceph-mon[33745]: pgmap v1302: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:23.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:38:22 smithi016 ceph-mon[37215]: pgmap v1302: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:23.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:38:22 smithi016 ceph-mon[29101]: pgmap v1302: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:23.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:38:23 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:38:24.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:38:23 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:38:24.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:38:23 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:38:24.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:38:23 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:38:25.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:38:24 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:38:25.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:38:24 smithi183 ceph-mon[33745]: pgmap v1303: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:25.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:38:24 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:38:25.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:38:24 smithi016 ceph-mon[37215]: pgmap v1303: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:25.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:38:24 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:38:25.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:38:24 smithi016 ceph-mon[29101]: pgmap v1303: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:27.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:38:26 smithi183 ceph-mon[33745]: pgmap v1304: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:27.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:38:26 smithi016 ceph-mon[37215]: pgmap v1304: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:27.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:38:26 smithi016 ceph-mon[29101]: pgmap v1304: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:29.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:38:28 smithi183 ceph-mon[33745]: pgmap v1305: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:29.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:38:28 smithi016 ceph-mon[29101]: pgmap v1305: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:29.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:38:28 smithi016 ceph-mon[37215]: pgmap v1305: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:29.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:38:28 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:38:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:38:31.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:38:30 smithi183 ceph-mon[33745]: pgmap v1306: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:31.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:38:30 smithi016 ceph-mon[29101]: pgmap v1306: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:31.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:38:30 smithi016 ceph-mon[37215]: pgmap v1306: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:33.035 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:38:32 smithi183 ceph-mon[33745]: pgmap v1307: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:33.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:38:32 smithi016 ceph-mon[29101]: pgmap v1307: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:33.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:38:32 smithi016 ceph-mon[37215]: pgmap v1307: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:33.291 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:38:33 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:38:35.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:38:34 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:38:35.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:38:34 smithi183 ceph-mon[33745]: pgmap v1308: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:35.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:38:34 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:38:35.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:38:34 smithi016 ceph-mon[29101]: pgmap v1308: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:35.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:38:34 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:38:35.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:38:34 smithi016 ceph-mon[37215]: pgmap v1308: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:37.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:38:36 smithi183 ceph-mon[33745]: pgmap v1309: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:37.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:38:36 smithi016 ceph-mon[29101]: pgmap v1309: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:37.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:38:36 smithi016 ceph-mon[37215]: pgmap v1309: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:39.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:38:38 smithi183 ceph-mon[33745]: pgmap v1310: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:39.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:38:38 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:38:39.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:38:38 smithi016 ceph-mon[29101]: pgmap v1310: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:39.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:38:38 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:38:39.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:38:38 smithi016 ceph-mon[37215]: pgmap v1310: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:39.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:38:38 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:38:39.163 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:38:38 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:38:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:38:41.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:38:40 smithi183 ceph-mon[33745]: pgmap v1311: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:41.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:38:40 smithi016 ceph-mon[29101]: pgmap v1311: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:41.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:38:40 smithi016 ceph-mon[37215]: pgmap v1311: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:43.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:38:42 smithi183 ceph-mon[33745]: pgmap v1312: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:43.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:38:42 smithi016 ceph-mon[29101]: pgmap v1312: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:43.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:38:42 smithi016 ceph-mon[37215]: pgmap v1312: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:43.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:38:43 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:38:45.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:38:44 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:38:45.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:38:44 smithi183 ceph-mon[33745]: pgmap v1313: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:45.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:38:44 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:38:45.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:38:44 smithi016 ceph-mon[29101]: pgmap v1313: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:45.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:38:44 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:38:45.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:38:44 smithi016 ceph-mon[37215]: pgmap v1313: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:47.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:38:46 smithi183 ceph-mon[33745]: pgmap v1314: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:38:47.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:38:46 smithi016 ceph-mon[37215]: pgmap v1314: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:38:47.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:38:46 smithi016 ceph-mon[29101]: pgmap v1314: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:38:48.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:38:47 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:38:48.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:38:47 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:38:48.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:38:47 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:38:48.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:38:47 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:38:48.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:38:47 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:38:48.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:38:47 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:38:48.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:38:47 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:38:48.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:38:47 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:38:48.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:38:47 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:38:48.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:38:47 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:38:48.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:38:47 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:38:48.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:38:47 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:38:49.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:38:48 smithi183 ceph-mon[33745]: pgmap v1315: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:49.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:38:48 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:38:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:38:49.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:38:48 smithi016 ceph-mon[37215]: pgmap v1315: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:49.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:38:48 smithi016 ceph-mon[29101]: pgmap v1315: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:51.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:38:50 smithi183 ceph-mon[33745]: pgmap v1316: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:38:51.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:38:50 smithi016 ceph-mon[37215]: pgmap v1316: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:38:51.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:38:50 smithi016 ceph-mon[29101]: pgmap v1316: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:38:52.041 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:38:51 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=cleanup t=2024-03-26T04:38:51.765727373Z level=info msg="Completed cleanup jobs" duration=2.38079ms 2024-03-26T04:38:53.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:38:52 smithi183 ceph-mon[33745]: pgmap v1317: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:53.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:38:52 smithi016 ceph-mon[37215]: pgmap v1317: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:53.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:38:52 smithi016 ceph-mon[29101]: pgmap v1317: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:38:53.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:38:53 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:38:54.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:38:53 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:38:54.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:38:53 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:38:54.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:38:53 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:38:55.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:38:54 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:38:55.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:38:54 smithi183 ceph-mon[33745]: pgmap v1318: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:55.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:38:54 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:38:55.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:38:54 smithi016 ceph-mon[29101]: pgmap v1318: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:55.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:38:54 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:38:55.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:38:54 smithi016 ceph-mon[37215]: pgmap v1318: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:57.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:38:56 smithi183 ceph-mon[33745]: pgmap v1319: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:38:57.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:38:56 smithi016 ceph-mon[37215]: pgmap v1319: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:38:57.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:38:56 smithi016 ceph-mon[29101]: pgmap v1319: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:38:59.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:38:58 smithi183 ceph-mon[33745]: pgmap v1320: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:59.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:38:58 smithi016 ceph-mon[37215]: pgmap v1320: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:59.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:38:58 smithi016 ceph-mon[29101]: pgmap v1320: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:38:59.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:38:58 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:38:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:39:00.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:38:59 smithi016 ceph-mon[37215]: pgmap v1321: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:00.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:38:59 smithi016 ceph-mon[29101]: pgmap v1321: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:00.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:38:59 smithi183 ceph-mon[33745]: pgmap v1321: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:02.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:39:02 smithi183 ceph-mon[33745]: pgmap v1322: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:02.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:39:02 smithi016 ceph-mon[29101]: pgmap v1322: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:02.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:39:02 smithi016 ceph-mon[37215]: pgmap v1322: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:03.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:39:03 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:39:04.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:39:04 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:39:04.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:39:04 smithi183 ceph-mon[33745]: pgmap v1323: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:04.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:39:04 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:39:04.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:39:04 smithi016 ceph-mon[29101]: pgmap v1323: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:04.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:39:04 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:39:04.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:39:04 smithi016 ceph-mon[37215]: pgmap v1323: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:06.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:39:06 smithi183 ceph-mon[33745]: pgmap v1324: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:39:06.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:39:06 smithi016 ceph-mon[29101]: pgmap v1324: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:39:06.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:39:06 smithi016 ceph-mon[37215]: pgmap v1324: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:39:08.745 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:39:08 smithi016 ceph-mon[37215]: pgmap v1325: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:08.746 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:39:08 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:39:08.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:39:08 smithi016 ceph-mon[29101]: pgmap v1325: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:08.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:39:08 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:39:08.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:39:08 smithi183 ceph-mon[33745]: pgmap v1325: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:08.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:39:08 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:39:09.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:39:08 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:39:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:39:10.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:39:10 smithi183 ceph-mon[33745]: pgmap v1326: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:39:10.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:39:10 smithi016 ceph-mon[37215]: pgmap v1326: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:39:10.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:39:10 smithi016 ceph-mon[29101]: pgmap v1326: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:39:12.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:39:12 smithi183 ceph-mon[33745]: pgmap v1327: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:12.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:39:12 smithi016 ceph-mon[37215]: pgmap v1327: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:12.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:39:12 smithi016 ceph-mon[29101]: pgmap v1327: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:13.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:39:13 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:39:14.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:39:14 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:39:14.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:39:14 smithi183 ceph-mon[33745]: pgmap v1328: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:14.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:39:14 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:39:14.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:39:14 smithi016 ceph-mon[37215]: pgmap v1328: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:14.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:39:14 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:39:14.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:39:14 smithi016 ceph-mon[29101]: pgmap v1328: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:16.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:39:16 smithi183 ceph-mon[33745]: pgmap v1329: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:39:16.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:39:16 smithi016 ceph-mon[37215]: pgmap v1329: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:39:16.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:39:16 smithi016 ceph-mon[29101]: pgmap v1329: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:39:18.745 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:39:18 smithi016 ceph-mon[29101]: pgmap v1330: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:18.745 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:39:18 smithi016 ceph-mon[37215]: pgmap v1330: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:18.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:39:18 smithi183 ceph-mon[33745]: pgmap v1330: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:19.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:39:18 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:39:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:39:20.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:39:20 smithi183 ceph-mon[33745]: pgmap v1331: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:20.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:39:20 smithi016 ceph-mon[29101]: pgmap v1331: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:20.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:39:20 smithi016 ceph-mon[37215]: pgmap v1331: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:22.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:39:22 smithi016 ceph-mon[29101]: pgmap v1332: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:22.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:39:22 smithi016 ceph-mon[37215]: pgmap v1332: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:22.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:39:22 smithi183 ceph-mon[33745]: pgmap v1332: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:23.541 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:39:23 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:39:23.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:39:23 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:39:23.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:39:23 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:39:23.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:39:23 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:39:24.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:39:24 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:39:24.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:39:24 smithi016 ceph-mon[29101]: pgmap v1333: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:24.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:39:24 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:39:24.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:39:24 smithi016 ceph-mon[37215]: pgmap v1333: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:24.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:39:24 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:39:24.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:39:24 smithi183 ceph-mon[33745]: pgmap v1333: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:26.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:39:26 smithi183 ceph-mon[33745]: pgmap v1334: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:26.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:39:26 smithi016 ceph-mon[29101]: pgmap v1334: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:26.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:39:26 smithi016 ceph-mon[37215]: pgmap v1334: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:28.745 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:39:28 smithi016 ceph-mon[29101]: pgmap v1335: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:28.746 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:39:28 smithi016 ceph-mon[37215]: pgmap v1335: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:28.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:39:28 smithi183 ceph-mon[33745]: pgmap v1335: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:29.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:39:28 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:39:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:39:30.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:39:30 smithi183 ceph-mon[33745]: pgmap v1336: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:39:30.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:39:30 smithi016 ceph-mon[29101]: pgmap v1336: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:39:30.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:39:30 smithi016 ceph-mon[37215]: pgmap v1336: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:39:32.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:39:32 smithi183 ceph-mon[33745]: pgmap v1337: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:32.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:39:32 smithi016 ceph-mon[37215]: pgmap v1337: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:32.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:39:32 smithi016 ceph-mon[29101]: pgmap v1337: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:33.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:39:33 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:39:34.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:39:34 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:39:34.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:39:34 smithi183 ceph-mon[33745]: pgmap v1338: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:34.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:39:34 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:39:34.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:39:34 smithi016 ceph-mon[37215]: pgmap v1338: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:34.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:39:34 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:39:34.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:39:34 smithi016 ceph-mon[29101]: pgmap v1338: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:36.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:39:36 smithi183 ceph-mon[33745]: pgmap v1339: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:39:36.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:39:36 smithi016 ceph-mon[37215]: pgmap v1339: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:39:36.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:39:36 smithi016 ceph-mon[29101]: pgmap v1339: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:39:38.745 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:39:38 smithi016 ceph-mon[37215]: pgmap v1340: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:38.745 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:39:38 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:39:38.745 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:39:38 smithi016 ceph-mon[29101]: pgmap v1340: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:38.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:39:38 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:39:38.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:39:38 smithi183 ceph-mon[33745]: pgmap v1340: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:38.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:39:38 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:39:39.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:39:38 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:39:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:39:40.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:39:40 smithi183 ceph-mon[33745]: pgmap v1341: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:39:40.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:39:40 smithi016 ceph-mon[29101]: pgmap v1341: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:39:40.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:39:40 smithi016 ceph-mon[37215]: pgmap v1341: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:39:42.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:39:42 smithi183 ceph-mon[33745]: pgmap v1342: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:42.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:39:42 smithi016 ceph-mon[29101]: pgmap v1342: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:42.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:39:42 smithi016 ceph-mon[37215]: pgmap v1342: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:39:43.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:39:43 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:39:44.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:39:44 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:39:44.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:39:44 smithi183 ceph-mon[33745]: pgmap v1343: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:44.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:39:44 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:39:44.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:39:44 smithi016 ceph-mon[29101]: pgmap v1343: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:44.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:39:44 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:39:44.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:39:44 smithi016 ceph-mon[37215]: pgmap v1343: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:46.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:39:46 smithi183 ceph-mon[33745]: pgmap v1344: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:39:46.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:39:46 smithi016 ceph-mon[29101]: pgmap v1344: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:39:46.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:39:46 smithi016 ceph-mon[37215]: pgmap v1344: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:39:48.745 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:39:48 smithi016 ceph-mon[37215]: pgmap v1345: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:48.745 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:39:48 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:39:48.745 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:39:48 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:48.745 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:39:48 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:39:48.745 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:39:48 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:39:48.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:39:48 smithi016 ceph-mon[29101]: pgmap v1345: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:48.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:39:48 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:39:48.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:39:48 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:48.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:39:48 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:39:48.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:39:48 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:39:48.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:39:48 smithi183 ceph-mon[33745]: pgmap v1345: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:48.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:39:48 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:39:48.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:39:48 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:39:48.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:39:48 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:39:48.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:39:48 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:39:49.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:39:48 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:39:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:39:50.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:39:50 smithi183 ceph-mon[33745]: pgmap v1346: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:39:50.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:39:50 smithi016 ceph-mon[37215]: pgmap v1346: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:39:50.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:39:50 smithi016 ceph-mon[29101]: pgmap v1346: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:39:52.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:39:52 smithi183 ceph-mon[33745]: pgmap v1347: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:39:52.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:39:52 smithi016 ceph-mon[37215]: pgmap v1347: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:39:52.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:39:52 smithi016 ceph-mon[29101]: pgmap v1347: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:39:53.487 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:39:53 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:39:53.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:39:53 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:39:53.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:39:53 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:39:53.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:39:53 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:39:54.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:39:54 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:39:54.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:39:54 smithi183 ceph-mon[33745]: pgmap v1348: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:54.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:39:54 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:39:54.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:39:54 smithi016 ceph-mon[37215]: pgmap v1348: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:54.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:39:54 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:39:54.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:39:54 smithi016 ceph-mon[29101]: pgmap v1348: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:56.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:39:56 smithi183 ceph-mon[33745]: pgmap v1349: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:39:56.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:39:56 smithi016 ceph-mon[37215]: pgmap v1349: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:39:56.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:39:56 smithi016 ceph-mon[29101]: pgmap v1349: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:39:58.746 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:39:58 smithi016 ceph-mon[37215]: pgmap v1350: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:58.746 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:39:58 smithi016 ceph-mon[29101]: pgmap v1350: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:58.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:39:58 smithi183 ceph-mon[33745]: pgmap v1350: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:39:59.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:39:58 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:39:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:40:00.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:00 smithi183 ceph-mon[33745]: pgmap v1351: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:40:00.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:00 smithi183 ceph-mon[33745]: Health detail: HEALTH_WARN 3 pool(s) do not have an application enabled 2024-03-26T04:40:00.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:00 smithi183 ceph-mon[33745]: [WRN] POOL_APP_NOT_ENABLED: 3 pool(s) do not have an application enabled 2024-03-26T04:40:00.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:00 smithi183 ceph-mon[33745]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-26T04:40:00.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:00 smithi183 ceph-mon[33745]: application not enabled on pool 'ListObjectssmithi016-75481-1' 2024-03-26T04:40:00.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:00 smithi183 ceph-mon[33745]: application not enabled on pool 'Limitssmithi016-75515-1' 2024-03-26T04:40:00.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:00 smithi183 ceph-mon[33745]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-26T04:40:00.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:00 smithi016 ceph-mon[29101]: pgmap v1351: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:40:00.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:00 smithi016 ceph-mon[29101]: Health detail: HEALTH_WARN 3 pool(s) do not have an application enabled 2024-03-26T04:40:00.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:00 smithi016 ceph-mon[29101]: [WRN] POOL_APP_NOT_ENABLED: 3 pool(s) do not have an application enabled 2024-03-26T04:40:00.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:00 smithi016 ceph-mon[29101]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-26T04:40:00.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:00 smithi016 ceph-mon[29101]: application not enabled on pool 'ListObjectssmithi016-75481-1' 2024-03-26T04:40:00.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:00 smithi016 ceph-mon[29101]: application not enabled on pool 'Limitssmithi016-75515-1' 2024-03-26T04:40:00.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:00 smithi016 ceph-mon[29101]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-26T04:40:00.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:00 smithi016 ceph-mon[37215]: pgmap v1351: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:40:00.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:00 smithi016 ceph-mon[37215]: Health detail: HEALTH_WARN 3 pool(s) do not have an application enabled 2024-03-26T04:40:00.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:00 smithi016 ceph-mon[37215]: [WRN] POOL_APP_NOT_ENABLED: 3 pool(s) do not have an application enabled 2024-03-26T04:40:00.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:00 smithi016 ceph-mon[37215]: application not enabled on pool 'ceph_test_rados_api_asio' 2024-03-26T04:40:00.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:00 smithi016 ceph-mon[37215]: application not enabled on pool 'ListObjectssmithi016-75481-1' 2024-03-26T04:40:00.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:00 smithi016 ceph-mon[37215]: application not enabled on pool 'Limitssmithi016-75515-1' 2024-03-26T04:40:00.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:00 smithi016 ceph-mon[37215]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-03-26T04:40:02.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:02 smithi183 ceph-mon[33745]: pgmap v1352: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:02.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:02 smithi016 ceph-mon[37215]: pgmap v1352: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:02.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:02 smithi016 ceph-mon[29101]: pgmap v1352: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:03.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:40:03 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:40:04.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:04 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:40:04.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:04 smithi183 ceph-mon[33745]: pgmap v1353: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:04.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:04 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:40:04.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:04 smithi016 ceph-mon[37215]: pgmap v1353: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:04.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:04 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:40:04.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:04 smithi016 ceph-mon[29101]: pgmap v1353: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:06.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:06 smithi183 ceph-mon[33745]: pgmap v1354: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:40:06.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:06 smithi016 ceph-mon[29101]: pgmap v1354: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:40:06.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:06 smithi016 ceph-mon[37215]: pgmap v1354: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:40:08.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:08 smithi183 ceph-mon[33745]: pgmap v1355: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:08.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:08 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:40:08.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:08 smithi016 ceph-mon[37215]: pgmap v1355: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:08.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:08 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:40:08.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:08 smithi016 ceph-mon[29101]: pgmap v1355: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:08.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:08 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:40:08.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:40:08 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:40:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:40:10.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:10 smithi183 ceph-mon[33745]: pgmap v1356: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:40:10.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:10 smithi016 ceph-mon[29101]: pgmap v1356: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:40:10.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:10 smithi016 ceph-mon[37215]: pgmap v1356: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:40:12.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:12 smithi183 ceph-mon[33745]: pgmap v1357: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:12.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:12 smithi016 ceph-mon[29101]: pgmap v1357: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:12.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:12 smithi016 ceph-mon[37215]: pgmap v1357: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:13.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:40:13 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:40:14.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:14 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:40:14.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:14 smithi183 ceph-mon[33745]: pgmap v1358: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:14.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:14 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:40:14.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:14 smithi016 ceph-mon[29101]: pgmap v1358: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:14.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:14 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:40:14.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:14 smithi016 ceph-mon[37215]: pgmap v1358: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:16.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:16 smithi183 ceph-mon[33745]: pgmap v1359: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:40:16.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:16 smithi016 ceph-mon[37215]: pgmap v1359: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:40:16.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:16 smithi016 ceph-mon[29101]: pgmap v1359: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:40:18.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:18 smithi183 ceph-mon[33745]: pgmap v1360: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:18.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:18 smithi016 ceph-mon[29101]: pgmap v1360: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:18.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:18 smithi016 ceph-mon[37215]: pgmap v1360: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:18.913 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:40:18 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:40:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:40:20.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:20 smithi183 ceph-mon[33745]: pgmap v1361: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:40:20.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:20 smithi016 ceph-mon[37215]: pgmap v1361: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:40:20.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:20 smithi016 ceph-mon[29101]: pgmap v1361: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:40:22.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:22 smithi183 ceph-mon[33745]: pgmap v1362: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:22.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:22 smithi016 ceph-mon[37215]: pgmap v1362: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:22.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:22 smithi016 ceph-mon[29101]: pgmap v1362: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:23.536 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:40:23 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:40:23.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:23 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:40:23.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:23 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:40:23.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:23 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:40:24.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:24 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:40:24.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:24 smithi183 ceph-mon[33745]: pgmap v1363: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:24.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:24 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:40:24.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:24 smithi016 ceph-mon[29101]: pgmap v1363: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:24.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:24 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:40:24.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:24 smithi016 ceph-mon[37215]: pgmap v1363: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:26.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:26 smithi183 ceph-mon[33745]: pgmap v1364: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:40:26.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:26 smithi016 ceph-mon[29101]: pgmap v1364: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:40:26.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:26 smithi016 ceph-mon[37215]: pgmap v1364: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:40:28.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:28 smithi016 ceph-mon[29101]: pgmap v1365: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:28.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:28 smithi016 ceph-mon[37215]: pgmap v1365: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:28.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:40:28 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:40:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:40:29.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:28 smithi183 ceph-mon[33745]: pgmap v1365: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:30.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:30 smithi016 ceph-mon[37215]: pgmap v1366: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:40:30.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:30 smithi016 ceph-mon[29101]: pgmap v1366: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:40:31.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:30 smithi183 ceph-mon[33745]: pgmap v1366: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:40:32.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:32 smithi016 ceph-mon[29101]: pgmap v1367: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:32.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:32 smithi016 ceph-mon[37215]: pgmap v1367: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:33.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:32 smithi183 ceph-mon[33745]: pgmap v1367: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:33.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:40:33 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:40:34.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:34 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:40:34.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:34 smithi016 ceph-mon[37215]: pgmap v1368: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:34.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:34 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:40:34.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:34 smithi016 ceph-mon[29101]: pgmap v1368: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:35.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:34 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:40:35.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:34 smithi183 ceph-mon[33745]: pgmap v1368: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:36.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:36 smithi016 ceph-mon[37215]: pgmap v1369: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:36.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:36 smithi016 ceph-mon[29101]: pgmap v1369: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:37.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:36 smithi183 ceph-mon[33745]: pgmap v1369: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:38.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:38 smithi016 ceph-mon[37215]: pgmap v1370: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:38.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:38 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:40:38.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:38 smithi016 ceph-mon[29101]: pgmap v1370: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:38.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:38 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:40:38.913 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:40:38 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:40:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:40:39.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:38 smithi183 ceph-mon[33745]: pgmap v1370: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:39.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:38 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:40:40.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:40 smithi016 ceph-mon[29101]: pgmap v1371: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:40.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:40 smithi016 ceph-mon[37215]: pgmap v1371: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:41.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:40 smithi183 ceph-mon[33745]: pgmap v1371: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:42.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:42 smithi016 ceph-mon[37215]: pgmap v1372: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:42.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:42 smithi016 ceph-mon[29101]: pgmap v1372: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:43.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:42 smithi183 ceph-mon[33745]: pgmap v1372: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:43.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:40:43 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:40:44.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:44 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:40:44.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:44 smithi016 ceph-mon[37215]: pgmap v1373: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:44.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:44 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:40:44.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:44 smithi016 ceph-mon[29101]: pgmap v1373: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:45.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:44 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:40:45.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:44 smithi183 ceph-mon[33745]: pgmap v1373: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:46.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:46 smithi016 ceph-mon[29101]: pgmap v1374: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:46.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:46 smithi016 ceph-mon[37215]: pgmap v1374: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:47.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:46 smithi183 ceph-mon[33745]: pgmap v1374: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:48.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:40:48 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:40:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:40:48.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:48 smithi016 ceph-mon[37215]: pgmap v1375: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:48.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:48 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:40:48.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:48 smithi016 ceph-mon[29101]: pgmap v1375: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:48.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:48 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:40:49.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:48 smithi183 ceph-mon[33745]: pgmap v1375: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:49.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:48 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:40:50.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:50 smithi016 ceph-mon[37215]: pgmap v1376: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:50.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:50 smithi016 ceph-mon[29101]: pgmap v1376: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:50.938 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:50 smithi183 ceph-mon[33745]: pgmap v1376: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:52.511 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:52 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:40:52.512 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:52 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:40:52.512 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:52 smithi183 ceph-mon[33745]: pgmap v1377: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:52.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:52 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:40:52.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:52 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:40:52.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:52 smithi016 ceph-mon[29101]: pgmap v1377: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:52.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:52 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:40:52.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:52 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:40:52.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:52 smithi016 ceph-mon[37215]: pgmap v1377: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:53.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:40:53 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:40:54.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:53 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:40:54.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:53 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:40:54.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:53 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:40:54.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:53 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:40:54.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:53 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:40:54.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:53 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:40:54.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:53 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:40:54.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:53 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:40:54.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:53 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:40:54.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:53 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:40:54.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:53 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:40:54.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:53 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:40:54.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:53 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:40:54.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:53 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:40:54.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:53 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:40:54.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:53 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:40:54.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:53 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:40:54.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:53 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:40:55.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:54 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:40:55.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:54 smithi183 ceph-mon[33745]: pgmap v1378: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:55.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:54 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:40:55.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:54 smithi016 ceph-mon[37215]: pgmap v1378: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:55.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:54 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:40:55.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:54 smithi016 ceph-mon[29101]: pgmap v1378: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:57.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:56 smithi183 ceph-mon[33745]: pgmap v1379: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:57.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:56 smithi016 ceph-mon[29101]: pgmap v1379: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:57.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:56 smithi016 ceph-mon[37215]: pgmap v1379: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:40:59.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:40:58 smithi183 ceph-mon[33745]: pgmap v1380: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:59.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:40:58 smithi016 ceph-mon[37215]: pgmap v1380: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:59.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:40:58 smithi016 ceph-mon[29101]: pgmap v1380: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:40:59.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:40:58 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:40:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:41:01.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:41:00 smithi183 ceph-mon[33745]: pgmap v1381: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:01.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:41:00 smithi016 ceph-mon[29101]: pgmap v1381: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:01.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:41:00 smithi016 ceph-mon[37215]: pgmap v1381: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:03.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:41:02 smithi183 ceph-mon[33745]: pgmap v1382: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:03.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:41:02 smithi016 ceph-mon[29101]: pgmap v1382: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:03.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:41:02 smithi016 ceph-mon[37215]: pgmap v1382: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:03.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:41:03 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:41:05.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:41:04 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:41:05.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:41:04 smithi183 ceph-mon[33745]: pgmap v1383: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:05.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:41:04 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:41:05.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:41:04 smithi016 ceph-mon[29101]: pgmap v1383: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:05.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:41:04 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:41:05.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:41:04 smithi016 ceph-mon[37215]: pgmap v1383: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:07.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:41:06 smithi183 ceph-mon[33745]: pgmap v1384: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:07.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:41:06 smithi016 ceph-mon[29101]: pgmap v1384: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:07.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:41:06 smithi016 ceph-mon[37215]: pgmap v1384: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:09.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:41:08 smithi183 ceph-mon[33745]: pgmap v1385: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:09.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:41:08 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:41:09.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:41:08 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:41:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:41:09.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:41:08 smithi016 ceph-mon[29101]: pgmap v1385: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:09.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:41:08 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:41:09.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:41:08 smithi016 ceph-mon[37215]: pgmap v1385: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:09.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:41:08 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:41:11.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:41:10 smithi183 ceph-mon[33745]: pgmap v1386: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:11.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:41:10 smithi016 ceph-mon[29101]: pgmap v1386: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:11.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:41:10 smithi016 ceph-mon[37215]: pgmap v1386: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:13.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:41:12 smithi183 ceph-mon[33745]: pgmap v1387: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:13.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:41:12 smithi016 ceph-mon[29101]: pgmap v1387: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:13.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:41:12 smithi016 ceph-mon[37215]: pgmap v1387: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:13.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:41:13 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:41:15.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:41:14 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:41:15.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:41:14 smithi183 ceph-mon[33745]: pgmap v1388: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:15.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:41:14 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:41:15.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:41:14 smithi016 ceph-mon[29101]: pgmap v1388: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:15.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:41:14 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:41:15.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:41:14 smithi016 ceph-mon[37215]: pgmap v1388: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:17.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:41:16 smithi183 ceph-mon[33745]: pgmap v1389: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:41:17.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:41:16 smithi016 ceph-mon[29101]: pgmap v1389: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:41:17.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:41:16 smithi016 ceph-mon[37215]: pgmap v1389: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:41:19.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:41:18 smithi183 ceph-mon[33745]: pgmap v1390: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:19.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:41:18 smithi016 ceph-mon[37215]: pgmap v1390: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:19.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:41:18 smithi016 ceph-mon[29101]: pgmap v1390: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:19.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:41:18 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:41:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:41:21.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:41:20 smithi183 ceph-mon[33745]: pgmap v1391: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:21.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:41:20 smithi016 ceph-mon[37215]: pgmap v1391: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:21.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:41:20 smithi016 ceph-mon[29101]: pgmap v1391: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:23.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:41:22 smithi183 ceph-mon[33745]: pgmap v1392: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:23.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:41:22 smithi016 ceph-mon[37215]: pgmap v1392: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:23.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:41:22 smithi016 ceph-mon[29101]: pgmap v1392: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:23.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:41:23 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:41:24.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:41:23 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:41:24.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:41:23 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:41:24.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:41:23 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:41:25.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:41:24 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:41:25.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:41:24 smithi183 ceph-mon[33745]: pgmap v1393: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:25.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:41:24 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:41:25.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:41:24 smithi016 ceph-mon[29101]: pgmap v1393: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:25.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:41:24 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:41:25.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:41:24 smithi016 ceph-mon[37215]: pgmap v1393: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:27.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:41:26 smithi183 ceph-mon[33745]: pgmap v1394: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:27.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:41:26 smithi016 ceph-mon[37215]: pgmap v1394: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:27.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:41:26 smithi016 ceph-mon[29101]: pgmap v1394: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:29.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:41:28 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:41:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:41:29.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:41:28 smithi016 ceph-mon[37215]: pgmap v1395: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:29.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:41:28 smithi016 ceph-mon[29101]: pgmap v1395: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:29.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:41:28 smithi183 ceph-mon[33745]: pgmap v1395: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:31.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:41:30 smithi016 ceph-mon[37215]: pgmap v1396: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:31.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:41:30 smithi016 ceph-mon[29101]: pgmap v1396: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:31.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:41:30 smithi183 ceph-mon[33745]: pgmap v1396: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:33.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:41:32 smithi016 ceph-mon[37215]: pgmap v1397: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:33.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:41:32 smithi016 ceph-mon[29101]: pgmap v1397: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:33.180 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:41:32 smithi183 ceph-mon[33745]: pgmap v1397: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:33.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:41:33 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:41:35.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:41:34 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:41:35.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:41:34 smithi016 ceph-mon[29101]: pgmap v1398: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:35.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:41:34 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:41:35.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:41:34 smithi016 ceph-mon[37215]: pgmap v1398: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:35.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:41:34 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:41:35.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:41:34 smithi183 ceph-mon[33745]: pgmap v1398: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:37.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:41:36 smithi016 ceph-mon[37215]: pgmap v1399: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:37.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:41:36 smithi016 ceph-mon[29101]: pgmap v1399: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:37.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:41:36 smithi183 ceph-mon[33745]: pgmap v1399: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:39.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:41:38 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:41:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:41:39.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:41:38 smithi016 ceph-mon[29101]: pgmap v1400: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:39.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:41:38 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:41:39.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:41:38 smithi016 ceph-mon[37215]: pgmap v1400: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:39.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:41:38 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:41:39.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:41:38 smithi183 ceph-mon[33745]: pgmap v1400: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:39.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:41:38 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:41:41.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:41:40 smithi016 ceph-mon[37215]: pgmap v1401: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:41.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:41:40 smithi016 ceph-mon[29101]: pgmap v1401: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:41.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:41:40 smithi183 ceph-mon[33745]: pgmap v1401: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:43.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:41:42 smithi016 ceph-mon[29101]: pgmap v1402: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:43.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:41:42 smithi016 ceph-mon[37215]: pgmap v1402: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:43.191 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:41:42 smithi183 ceph-mon[33745]: pgmap v1402: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:43.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:41:43 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:41:45.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:41:44 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:41:45.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:41:44 smithi016 ceph-mon[29101]: pgmap v1403: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:45.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:41:44 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:41:45.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:41:44 smithi016 ceph-mon[37215]: pgmap v1403: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:45.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:41:44 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:41:45.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:41:44 smithi183 ceph-mon[33745]: pgmap v1403: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:47.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:41:46 smithi016 ceph-mon[29101]: pgmap v1404: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:47.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:41:46 smithi016 ceph-mon[37215]: pgmap v1404: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:47.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:41:46 smithi183 ceph-mon[33745]: pgmap v1404: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:48.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:41:47 smithi016 ceph-mon[29101]: pgmap v1405: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:48.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:41:47 smithi016 ceph-mon[37215]: pgmap v1405: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:48.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:41:47 smithi183 ceph-mon[33745]: pgmap v1405: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:49.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:41:48 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:41:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:41:50.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:41:50 smithi183 ceph-mon[33745]: pgmap v1406: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:50.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:41:50 smithi016 ceph-mon[29101]: pgmap v1406: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:50.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:41:50 smithi016 ceph-mon[37215]: pgmap v1406: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:52.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:41:52 smithi183 ceph-mon[33745]: pgmap v1407: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:52.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:41:52 smithi016 ceph-mon[29101]: pgmap v1407: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:52.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:41:52 smithi016 ceph-mon[37215]: pgmap v1407: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:53.507 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:41:53 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:41:53.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:41:53 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:41:53.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:41:53 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:41:53.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:41:53 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:41:53.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:41:53 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:41:53.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:41:53 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:41:53.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:41:53 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:41:54.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:41:54 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:41:54.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:41:54 smithi183 ceph-mon[33745]: pgmap v1408: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:54.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:41:54 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:54.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:41:54 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:41:54.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:41:54 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:41:54.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:41:54 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:41:54.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:41:54 smithi016 ceph-mon[37215]: pgmap v1408: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:54.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:41:54 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:54.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:41:54 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:41:54.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:41:54 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:41:54.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:41:54 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:41:54.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:41:54 smithi016 ceph-mon[29101]: pgmap v1408: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:54.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:41:54 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:41:54.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:41:54 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:41:54.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:41:54 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:41:56.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:41:56 smithi183 ceph-mon[33745]: pgmap v1409: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:56.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:41:56 smithi016 ceph-mon[29101]: pgmap v1409: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:56.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:41:56 smithi016 ceph-mon[37215]: pgmap v1409: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:41:58.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:41:58 smithi183 ceph-mon[33745]: pgmap v1410: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:58.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:41:58 smithi016 ceph-mon[37215]: pgmap v1410: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:58.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:41:58 smithi016 ceph-mon[29101]: pgmap v1410: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:41:58.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:41:58 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:41:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:42:00.794 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:42:00 smithi183 ceph-mon[33745]: pgmap v1411: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:42:00.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:42:00 smithi016 ceph-mon[37215]: pgmap v1411: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:42:00.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:42:00 smithi016 ceph-mon[29101]: pgmap v1411: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:42:02.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:42:02 smithi183 ceph-mon[33745]: pgmap v1412: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:42:02.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:42:02 smithi016 ceph-mon[37215]: pgmap v1412: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:42:02.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:42:02 smithi016 ceph-mon[29101]: pgmap v1412: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:42:03.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:42:03 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:42:04.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:42:04 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:42:04.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:42:04 smithi183 ceph-mon[33745]: pgmap v1413: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:42:04.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:42:04 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:42:04.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:42:04 smithi016 ceph-mon[37215]: pgmap v1413: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:42:04.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:42:04 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:42:04.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:42:04 smithi016 ceph-mon[29101]: pgmap v1413: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:42:06.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:42:06 smithi183 ceph-mon[33745]: pgmap v1414: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:42:06.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:42:06 smithi016 ceph-mon[37215]: pgmap v1414: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:42:06.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:42:06 smithi016 ceph-mon[29101]: pgmap v1414: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:42:08.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:42:08 smithi183 ceph-mon[33745]: pgmap v1415: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:42:08.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:42:08 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:42:08.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:42:08 smithi016 ceph-mon[37215]: pgmap v1415: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:42:08.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:42:08 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:42:08.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:42:08 smithi016 ceph-mon[29101]: pgmap v1415: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:42:08.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:42:08 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:42:08.913 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:42:08 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:42:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:42:10.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:42:10 smithi183 ceph-mon[33745]: pgmap v1416: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:42:10.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:42:10 smithi016 ceph-mon[29101]: pgmap v1416: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:42:10.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:42:10 smithi016 ceph-mon[37215]: pgmap v1416: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:42:12.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:42:12 smithi183 ceph-mon[33745]: pgmap v1417: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:42:12.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:42:12 smithi016 ceph-mon[37215]: pgmap v1417: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:42:12.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:42:12 smithi016 ceph-mon[29101]: pgmap v1417: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:42:13.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:42:13 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:42:14.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:42:14 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:42:14.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:42:14 smithi183 ceph-mon[33745]: pgmap v1418: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:42:14.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:42:14 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:42:14.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:42:14 smithi016 ceph-mon[37215]: pgmap v1418: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:42:14.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:42:14 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:42:14.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:42:14 smithi016 ceph-mon[29101]: pgmap v1418: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:42:16.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:42:16 smithi183 ceph-mon[33745]: pgmap v1419: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:42:16.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:42:16 smithi016 ceph-mon[37215]: pgmap v1419: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:42:16.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:42:16 smithi016 ceph-mon[29101]: pgmap v1419: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:42:18.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:42:18 smithi016 ceph-mon[37215]: pgmap v1420: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:42:18.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:42:18 smithi016 ceph-mon[29101]: pgmap v1420: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:42:18.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:42:18 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:42:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:42:19.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:42:18 smithi183 ceph-mon[33745]: pgmap v1420: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:42:20.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:42:20 smithi016 ceph-mon[37215]: pgmap v1421: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:42:20.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:42:20 smithi016 ceph-mon[29101]: pgmap v1421: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:42:21.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:42:20 smithi183 ceph-mon[33745]: pgmap v1421: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:42:22.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:42:22 smithi016 ceph-mon[37215]: pgmap v1422: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:42:22.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:42:22 smithi016 ceph-mon[29101]: pgmap v1422: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:42:23.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:42:22 smithi183 ceph-mon[33745]: pgmap v1422: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:42:23.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:42:23 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:42:23.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:42:23 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:42:23.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:42:23 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:42:24.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:42:23 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:42:24.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:42:24 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:42:24.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:42:24 smithi016 ceph-mon[37215]: pgmap v1423: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:42:24.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:42:24 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:42:24.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:42:24 smithi016 ceph-mon[29101]: pgmap v1423: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:42:25.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:42:24 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:42:25.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:42:24 smithi183 ceph-mon[33745]: pgmap v1423: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:42:26.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:42:26 smithi016 ceph-mon[37215]: pgmap v1424: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:42:26.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:42:26 smithi016 ceph-mon[29101]: pgmap v1424: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:42:27.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:42:26 smithi183 ceph-mon[33745]: pgmap v1424: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:42:28.843 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:42:28 smithi183 ceph-mon[33745]: pgmap v1425: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:42:28.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:42:28 smithi016 ceph-mon[29101]: pgmap v1425: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:42:28.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:42:28 smithi016 ceph-mon[37215]: pgmap v1425: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:42:28.913 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:42:28 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:42:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:42:30.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:42:30 smithi016 ceph-mon[37215]: pgmap v1426: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:42:30.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:42:30 smithi016 ceph-mon[29101]: pgmap v1426: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:42:31.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:42:30 smithi183 ceph-mon[33745]: pgmap v1426: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:42:32.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:42:32 smithi016 ceph-mon[37215]: pgmap v1427: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:42:32.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:42:32 smithi016 ceph-mon[29101]: pgmap v1427: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:42:33.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:42:32 smithi183 ceph-mon[33745]: pgmap v1427: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:42:33.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:42:33 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:42:34.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:42:34 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:42:34.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:42:34 smithi016 ceph-mon[37215]: pgmap v1428: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:42:34.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:42:34 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:42:34.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:42:34 smithi016 ceph-mon[29101]: pgmap v1428: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:42:35.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:42:34 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:42:35.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:42:34 smithi183 ceph-mon[33745]: pgmap v1428: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:42:36.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:42:36 smithi016 ceph-mon[37215]: pgmap v1429: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:42:36.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:42:36 smithi016 ceph-mon[29101]: pgmap v1429: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:42:37.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:42:36 smithi183 ceph-mon[33745]: pgmap v1429: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:42:38.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:42:38 smithi016 ceph-mon[29101]: pgmap v1430: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:42:38.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:42:38 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:42:38.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:42:38 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:42:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:42:38.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:42:38 smithi016 ceph-mon[37215]: pgmap v1430: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:42:38.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:42:38 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:42:39.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:42:38 smithi183 ceph-mon[33745]: pgmap v1430: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:42:39.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:42:38 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:42:40.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:42:40 smithi016 ceph-mon[29101]: pgmap v1431: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:42:40.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:42:40 smithi016 ceph-mon[37215]: pgmap v1431: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:42:41.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:42:40 smithi183 ceph-mon[33745]: pgmap v1431: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:42:42.914 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:42:42 smithi016 ceph-mon[37215]: pgmap v1432: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:42:42.914 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:42:42 smithi016 ceph-mon[29101]: pgmap v1432: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:42:43.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:42:42 smithi183 ceph-mon[33745]: pgmap v1432: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:42:43.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:42:43 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:42:44.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:42:44 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:42:44.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:42:44 smithi016 ceph-mon[37215]: pgmap v1433: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:42:44.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:42:44 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:42:44.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:42:44 smithi016 ceph-mon[29101]: pgmap v1433: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:42:45.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:42:44 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:42:45.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:42:44 smithi183 ceph-mon[33745]: pgmap v1433: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:42:46.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:42:46 smithi016 ceph-mon[37215]: pgmap v1434: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:42:46.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:42:46 smithi016 ceph-mon[29101]: pgmap v1434: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:42:47.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:42:46 smithi183 ceph-mon[33745]: pgmap v1434: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:42:48.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:42:48 smithi016 ceph-mon[37215]: pgmap v1435: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:42:48.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:42:48 smithi016 ceph-mon[29101]: pgmap v1435: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:42:48.913 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:42:48 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:42:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:42:49.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:42:48 smithi183 ceph-mon[33745]: pgmap v1435: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:42:50.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:42:50 smithi016 ceph-mon[37215]: pgmap v1436: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:42:50.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:42:50 smithi016 ceph-mon[29101]: pgmap v1436: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:42:51.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:42:50 smithi183 ceph-mon[33745]: pgmap v1436: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:42:52.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:42:52 smithi016 ceph-mon[37215]: pgmap v1437: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:42:52.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:42:52 smithi016 ceph-mon[29101]: pgmap v1437: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:42:53.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:42:52 smithi183 ceph-mon[33745]: pgmap v1437: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:42:53.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:42:53 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:42:53.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:42:53 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:42:53.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:42:53 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:42:54.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:42:53 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:42:54.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:42:54 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:42:54.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:42:54 smithi016 ceph-mon[37215]: pgmap v1438: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:42:54.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:42:54 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:42:54.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:42:54 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:42:54.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:42:54 smithi016 ceph-mon[29101]: pgmap v1438: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:42:54.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:42:54 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:42:55.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:42:54 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:42:55.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:42:54 smithi183 ceph-mon[33745]: pgmap v1438: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:42:55.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:42:54 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:42:55.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:42:55 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:55.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:42:55 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:42:55.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:42:55 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:42:55.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:42:55 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:55.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:42:55 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:42:55.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:42:55 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:42:56.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:42:55 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:42:56.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:42:55 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:42:56.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:42:55 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:42:57.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:42:56 smithi183 ceph-mon[33745]: pgmap v1439: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:42:57.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:42:56 smithi016 ceph-mon[29101]: pgmap v1439: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:42:57.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:42:56 smithi016 ceph-mon[37215]: pgmap v1439: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:42:59.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:42:58 smithi183 ceph-mon[33745]: pgmap v1440: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:42:59.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:42:58 smithi016 ceph-mon[29101]: pgmap v1440: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:42:59.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:42:58 smithi016 ceph-mon[37215]: pgmap v1440: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:42:59.163 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:42:58 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:42:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:43:01.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:00 smithi183 ceph-mon[33745]: pgmap v1441: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:43:01.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:00 smithi016 ceph-mon[29101]: pgmap v1441: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:43:01.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:00 smithi016 ceph-mon[37215]: pgmap v1441: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:43:03.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:02 smithi183 ceph-mon[33745]: pgmap v1442: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:43:03.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:02 smithi016 ceph-mon[29101]: pgmap v1442: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:43:03.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:02 smithi016 ceph-mon[37215]: pgmap v1442: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:43:03.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:43:03 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:43:05.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:04 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:43:05.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:04 smithi183 ceph-mon[33745]: pgmap v1443: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:05.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:04 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:43:05.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:04 smithi016 ceph-mon[29101]: pgmap v1443: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:05.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:04 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:43:05.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:04 smithi016 ceph-mon[37215]: pgmap v1443: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:07.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:06 smithi183 ceph-mon[33745]: pgmap v1444: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:43:07.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:06 smithi016 ceph-mon[29101]: pgmap v1444: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:43:07.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:06 smithi016 ceph-mon[37215]: pgmap v1444: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:43:09.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:08 smithi183 ceph-mon[33745]: pgmap v1445: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:09.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:08 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:43:09.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:43:08 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:43:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:43:09.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:08 smithi016 ceph-mon[29101]: pgmap v1445: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:09.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:08 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:43:09.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:08 smithi016 ceph-mon[37215]: pgmap v1445: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:09.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:08 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:43:11.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:10 smithi183 ceph-mon[33745]: pgmap v1446: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:43:11.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:10 smithi016 ceph-mon[29101]: pgmap v1446: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:43:11.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:10 smithi016 ceph-mon[37215]: pgmap v1446: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:43:13.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:12 smithi183 ceph-mon[33745]: pgmap v1447: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:13.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:12 smithi016 ceph-mon[29101]: pgmap v1447: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:13.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:12 smithi016 ceph-mon[37215]: pgmap v1447: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:13.541 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:43:13 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:43:15.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:14 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:43:15.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:14 smithi183 ceph-mon[33745]: pgmap v1448: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:15.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:14 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:43:15.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:14 smithi016 ceph-mon[29101]: pgmap v1448: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:15.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:14 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:43:15.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:14 smithi016 ceph-mon[37215]: pgmap v1448: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:17.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:16 smithi183 ceph-mon[33745]: pgmap v1449: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:43:17.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:16 smithi016 ceph-mon[29101]: pgmap v1449: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:43:17.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:16 smithi016 ceph-mon[37215]: pgmap v1449: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:43:19.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:18 smithi016 ceph-mon[37215]: pgmap v1450: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:19.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:18 smithi016 ceph-mon[29101]: pgmap v1450: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:19.163 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:43:18 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:43:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:43:19.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:18 smithi183 ceph-mon[33745]: pgmap v1450: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:21.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:20 smithi016 ceph-mon[37215]: pgmap v1451: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:43:21.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:20 smithi016 ceph-mon[29101]: pgmap v1451: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:43:21.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:20 smithi183 ceph-mon[33745]: pgmap v1451: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:43:23.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:22 smithi016 ceph-mon[37215]: pgmap v1452: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:43:23.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:22 smithi016 ceph-mon[29101]: pgmap v1452: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:43:23.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:22 smithi183 ceph-mon[33745]: pgmap v1452: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:43:23.791 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:43:23 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:43:24.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:23 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:43:24.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:23 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:43:24.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:23 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:43:25.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:24 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:43:25.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:24 smithi016 ceph-mon[37215]: pgmap v1453: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:25.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:24 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:43:25.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:24 smithi016 ceph-mon[29101]: pgmap v1453: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:25.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:24 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:43:25.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:24 smithi183 ceph-mon[33745]: pgmap v1453: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:27.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:26 smithi016 ceph-mon[37215]: pgmap v1454: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:43:27.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:26 smithi016 ceph-mon[29101]: pgmap v1454: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:43:27.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:26 smithi183 ceph-mon[33745]: pgmap v1454: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:43:29.093 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:28 smithi183 ceph-mon[33745]: pgmap v1455: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:29.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:28 smithi016 ceph-mon[37215]: pgmap v1455: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:29.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:28 smithi016 ceph-mon[29101]: pgmap v1455: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:29.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:43:28 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:43:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:43:31.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:30 smithi016 ceph-mon[37215]: pgmap v1456: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:43:31.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:30 smithi016 ceph-mon[29101]: pgmap v1456: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:43:31.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:30 smithi183 ceph-mon[33745]: pgmap v1456: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:43:33.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:32 smithi016 ceph-mon[29101]: pgmap v1457: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:33.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:32 smithi016 ceph-mon[37215]: pgmap v1457: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:33.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:32 smithi183 ceph-mon[33745]: pgmap v1457: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:33.791 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:43:33 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:43:35.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:34 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:43:35.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:34 smithi016 ceph-mon[29101]: pgmap v1458: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:35.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:34 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:43:35.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:34 smithi016 ceph-mon[37215]: pgmap v1458: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:35.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:34 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:43:35.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:34 smithi183 ceph-mon[33745]: pgmap v1458: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:37.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:36 smithi016 ceph-mon[29101]: pgmap v1459: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:43:37.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:36 smithi016 ceph-mon[37215]: pgmap v1459: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:43:37.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:36 smithi183 ceph-mon[33745]: pgmap v1459: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:43:39.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:43:38 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:43:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:43:39.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:38 smithi016 ceph-mon[29101]: pgmap v1460: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:39.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:38 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:43:39.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:38 smithi016 ceph-mon[37215]: pgmap v1460: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:39.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:38 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:43:39.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:38 smithi183 ceph-mon[33745]: pgmap v1460: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:39.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:38 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:43:41.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:40 smithi016 ceph-mon[29101]: pgmap v1461: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T04:43:41.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:40 smithi016 ceph-mon[37215]: pgmap v1461: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T04:43:41.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:40 smithi183 ceph-mon[33745]: pgmap v1461: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T04:43:43.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:42 smithi016 ceph-mon[29101]: pgmap v1462: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T04:43:43.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:42 smithi016 ceph-mon[37215]: pgmap v1462: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T04:43:43.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:42 smithi183 ceph-mon[33745]: pgmap v1462: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T04:43:43.791 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:43:43 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:43:45.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:44 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:43:45.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:44 smithi016 ceph-mon[29101]: pgmap v1463: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:45.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:44 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:43:45.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:44 smithi016 ceph-mon[37215]: pgmap v1463: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:45.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:44 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:43:45.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:44 smithi183 ceph-mon[33745]: pgmap v1463: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:47.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:46 smithi016 ceph-mon[29101]: pgmap v1464: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T04:43:47.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:46 smithi016 ceph-mon[37215]: pgmap v1464: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T04:43:47.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:46 smithi183 ceph-mon[33745]: pgmap v1464: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T04:43:48.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:47 smithi016 ceph-mon[29101]: pgmap v1465: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:48.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:47 smithi016 ceph-mon[37215]: pgmap v1465: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:48.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:47 smithi183 ceph-mon[33745]: pgmap v1465: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:49.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:43:48 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:43:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:43:50.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:50 smithi016 ceph-mon[29101]: pgmap v1466: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:43:50.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:50 smithi016 ceph-mon[37215]: pgmap v1466: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:43:51.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:50 smithi183 ceph-mon[33745]: pgmap v1466: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:43:52.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:52 smithi016 ceph-mon[37215]: pgmap v1467: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:52.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:52 smithi016 ceph-mon[29101]: pgmap v1467: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:53.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:52 smithi183 ceph-mon[33745]: pgmap v1467: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:43:53.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:53 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:43:53.791 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:43:53 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:43:53.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:53 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:43:53.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:53 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:43:54.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:54 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:43:54.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:54 smithi016 ceph-mon[29101]: pgmap v1468: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:54.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:54 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:43:54.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:54 smithi016 ceph-mon[37215]: pgmap v1468: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:55.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:54 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:43:55.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:54 smithi183 ceph-mon[33745]: pgmap v1468: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:55.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:55 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:43:55.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:55 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:43:55.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:55 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:43:55.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:55 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:43:55.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:55 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:43:55.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:55 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:43:55.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:55 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:43:55.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:55 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:43:55.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:55 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:43:55.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:55 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:43:56.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:55 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:43:56.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:55 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:43:56.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:55 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:43:56.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:55 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:43:56.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:55 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:43:56.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:56 smithi016 ceph-mon[37215]: pgmap v1469: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:43:56.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:56 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:56.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:56 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:43:56.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:56 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:43:56.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:56 smithi016 ceph-mon[29101]: pgmap v1469: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:43:56.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:56 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:56.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:56 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:43:56.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:56 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:43:57.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:56 smithi183 ceph-mon[33745]: pgmap v1469: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:43:57.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:56 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:43:57.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:56 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:43:57.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:56 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:43:58.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:43:58 smithi016 ceph-mon[37215]: pgmap v1470: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:58.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:43:58 smithi016 ceph-mon[29101]: pgmap v1470: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:43:58.913 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:43:58 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:43:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:43:59.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:43:58 smithi183 ceph-mon[33745]: pgmap v1470: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:00.843 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:44:00 smithi183 ceph-mon[33745]: pgmap v1471: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:00.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:44:00 smithi016 ceph-mon[37215]: pgmap v1471: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:00.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:44:00 smithi016 ceph-mon[29101]: pgmap v1471: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:02.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:44:02 smithi016 ceph-mon[37215]: pgmap v1472: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:02.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:44:02 smithi016 ceph-mon[29101]: pgmap v1472: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:03.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:44:02 smithi183 ceph-mon[33745]: pgmap v1472: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:03.791 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:44:03 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:44:04.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:44:04 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:44:04.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:44:04 smithi016 ceph-mon[37215]: pgmap v1473: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:04.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:44:04 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:44:04.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:44:04 smithi016 ceph-mon[29101]: pgmap v1473: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:05.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:44:04 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:44:05.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:44:04 smithi183 ceph-mon[33745]: pgmap v1473: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:06.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:44:06 smithi016 ceph-mon[37215]: pgmap v1474: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:06.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:44:06 smithi016 ceph-mon[29101]: pgmap v1474: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:07.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:44:06 smithi183 ceph-mon[33745]: pgmap v1474: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:08.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:44:08 smithi016 ceph-mon[37215]: pgmap v1475: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:08.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:44:08 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:44:08.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:44:08 smithi016 ceph-mon[29101]: pgmap v1475: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:08.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:44:08 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:44:08.913 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:44:08 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:44:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:44:09.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:44:08 smithi183 ceph-mon[33745]: pgmap v1475: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:09.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:44:08 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:44:10.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:44:10 smithi016 ceph-mon[29101]: pgmap v1476: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:44:10.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:44:10 smithi016 ceph-mon[37215]: pgmap v1476: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:44:11.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:44:10 smithi183 ceph-mon[33745]: pgmap v1476: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:44:12.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:44:12 smithi016 ceph-mon[37215]: pgmap v1477: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:12.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:44:12 smithi016 ceph-mon[29101]: pgmap v1477: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:13.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:44:12 smithi183 ceph-mon[33745]: pgmap v1477: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:13.791 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:44:13 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:44:14.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:44:14 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:44:14.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:44:14 smithi016 ceph-mon[37215]: pgmap v1478: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:14.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:44:14 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:44:14.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:44:14 smithi016 ceph-mon[29101]: pgmap v1478: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:15.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:44:14 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:44:15.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:44:14 smithi183 ceph-mon[33745]: pgmap v1478: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:16.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:44:16 smithi016 ceph-mon[37215]: pgmap v1479: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:44:16.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:44:16 smithi016 ceph-mon[29101]: pgmap v1479: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:44:17.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:44:16 smithi183 ceph-mon[33745]: pgmap v1479: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:44:18.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:44:18 smithi016 ceph-mon[29101]: pgmap v1480: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:18.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:44:18 smithi016 ceph-mon[37215]: pgmap v1480: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:18.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:44:18 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:44:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:44:19.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:44:18 smithi183 ceph-mon[33745]: pgmap v1480: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:20.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:44:20 smithi016 ceph-mon[37215]: pgmap v1481: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:44:20.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:44:20 smithi016 ceph-mon[29101]: pgmap v1481: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:44:21.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:44:20 smithi183 ceph-mon[33745]: pgmap v1481: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:44:22.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:44:22 smithi016 ceph-mon[29101]: pgmap v1482: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:22.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:44:22 smithi016 ceph-mon[37215]: pgmap v1482: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:23.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:44:22 smithi183 ceph-mon[33745]: pgmap v1482: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:23.602 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:44:23 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:44:23.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:44:23 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:44:23.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:44:23 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:44:24.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:44:23 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:44:24.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:44:24 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:44:24.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:44:24 smithi016 ceph-mon[29101]: pgmap v1483: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:24.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:44:24 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:44:24.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:44:24 smithi016 ceph-mon[37215]: pgmap v1483: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:25.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:44:24 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:44:25.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:44:24 smithi183 ceph-mon[33745]: pgmap v1483: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:26.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:44:26 smithi016 ceph-mon[29101]: pgmap v1484: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:44:26.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:44:26 smithi016 ceph-mon[37215]: pgmap v1484: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:44:27.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:44:26 smithi183 ceph-mon[33745]: pgmap v1484: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:44:28.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:44:28 smithi016 ceph-mon[29101]: pgmap v1485: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:28.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:44:28 smithi016 ceph-mon[37215]: pgmap v1485: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:28.913 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:44:28 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:44:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:44:29.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:44:28 smithi183 ceph-mon[33745]: pgmap v1485: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:30.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:44:30 smithi016 ceph-mon[29101]: pgmap v1486: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:44:30.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:44:30 smithi016 ceph-mon[37215]: pgmap v1486: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:44:31.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:44:30 smithi183 ceph-mon[33745]: pgmap v1486: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:44:32.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:44:32 smithi016 ceph-mon[29101]: pgmap v1487: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:32.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:44:32 smithi016 ceph-mon[37215]: pgmap v1487: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:33.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:44:32 smithi183 ceph-mon[33745]: pgmap v1487: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:33.791 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:44:33 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:44:34.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:44:34 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:44:34.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:44:34 smithi016 ceph-mon[37215]: pgmap v1488: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:34.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:44:34 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:44:34.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:44:34 smithi016 ceph-mon[29101]: pgmap v1488: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:35.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:44:34 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:44:35.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:44:34 smithi183 ceph-mon[33745]: pgmap v1488: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:36.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:44:36 smithi016 ceph-mon[37215]: pgmap v1489: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:44:36.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:44:36 smithi016 ceph-mon[29101]: pgmap v1489: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:44:37.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:44:36 smithi183 ceph-mon[33745]: pgmap v1489: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:44:38.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:44:38 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:44:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:44:38.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:44:38 smithi016 ceph-mon[29101]: pgmap v1490: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:38.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:44:38 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:44:38.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:44:38 smithi016 ceph-mon[37215]: pgmap v1490: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:38.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:44:38 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:44:39.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:44:38 smithi183 ceph-mon[33745]: pgmap v1490: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:39.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:44:38 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:44:40.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:44:40 smithi016 ceph-mon[29101]: pgmap v1491: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:44:40.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:44:40 smithi016 ceph-mon[37215]: pgmap v1491: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:44:41.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:44:40 smithi183 ceph-mon[33745]: pgmap v1491: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:44:42.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:44:42 smithi016 ceph-mon[29101]: pgmap v1492: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:42.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:44:42 smithi016 ceph-mon[37215]: pgmap v1492: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:43.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:44:42 smithi183 ceph-mon[33745]: pgmap v1492: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:43.791 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:44:43 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:44:44.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:44:44 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:44:44.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:44:44 smithi016 ceph-mon[37215]: pgmap v1493: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:44.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:44:44 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:44:44.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:44:44 smithi016 ceph-mon[29101]: pgmap v1493: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:45.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:44:44 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:44:45.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:44:44 smithi183 ceph-mon[33745]: pgmap v1493: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:46.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:44:46 smithi016 ceph-mon[37215]: pgmap v1494: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:44:46.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:44:46 smithi016 ceph-mon[29101]: pgmap v1494: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:44:47.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:44:46 smithi183 ceph-mon[33745]: pgmap v1494: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:44:48.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:44:48 smithi016 ceph-mon[29101]: pgmap v1495: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:48.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:44:48 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:44:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:44:48.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:44:48 smithi016 ceph-mon[37215]: pgmap v1495: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:49.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:44:48 smithi183 ceph-mon[33745]: pgmap v1495: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:50.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:44:50 smithi016 ceph-mon[37215]: pgmap v1496: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:50.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:44:50 smithi016 ceph-mon[29101]: pgmap v1496: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:51.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:44:50 smithi183 ceph-mon[33745]: pgmap v1496: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:52.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:44:52 smithi016 ceph-mon[29101]: pgmap v1497: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:52.662 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:44:52 smithi016 ceph-mon[37215]: pgmap v1497: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:52.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:44:52 smithi183 ceph-mon[33745]: pgmap v1497: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:53.791 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:44:53 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:44:54.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:44:54 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:44:54.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:44:54 smithi016 ceph-mon[37215]: pgmap v1498: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:54.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:44:54 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:44:54.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:44:54 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:44:54.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:44:54 smithi016 ceph-mon[29101]: pgmap v1498: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:54.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:44:54 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:44:55.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:44:54 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:44:55.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:44:54 smithi183 ceph-mon[33745]: pgmap v1498: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:55.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:44:54 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:44:56.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:44:56 smithi016 ceph-mon[29101]: pgmap v1499: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:56.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:44:56 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:44:56.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:44:56 smithi016 ceph-mon[37215]: pgmap v1499: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:56.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:44:56 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:44:57.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:44:56 smithi183 ceph-mon[33745]: pgmap v1499: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:44:57.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:44:56 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:44:57.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:44:57 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:44:57.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:44:57 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:44:57.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:44:57 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:44:57.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:44:57 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:44:57.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:44:57 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:44:57.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:44:57 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:44:58.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:44:57 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:44:58.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:44:57 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:44:58.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:44:57 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:44:59.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:44:58 smithi183 ceph-mon[33745]: pgmap v1500: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:59.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:44:58 smithi016 ceph-mon[37215]: pgmap v1500: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:44:59.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:44:58 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:44:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:44:59.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:44:58 smithi016 ceph-mon[29101]: pgmap v1500: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:01.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:45:00 smithi183 ceph-mon[33745]: pgmap v1501: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:01.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:45:00 smithi016 ceph-mon[37215]: pgmap v1501: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:01.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:45:00 smithi016 ceph-mon[29101]: pgmap v1501: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:03.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:45:02 smithi183 ceph-mon[33745]: pgmap v1502: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:03.166 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:45:02 smithi016 ceph-mon[37215]: pgmap v1502: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:03.166 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:45:02 smithi016 ceph-mon[29101]: pgmap v1502: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:03.791 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:45:03 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:45:05.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:45:04 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:45:05.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:45:04 smithi183 ceph-mon[33745]: pgmap v1503: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:05.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:45:04 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:45:05.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:45:04 smithi016 ceph-mon[37215]: pgmap v1503: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:05.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:45:04 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:45:05.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:45:04 smithi016 ceph-mon[29101]: pgmap v1503: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:07.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:45:06 smithi183 ceph-mon[33745]: pgmap v1504: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:45:07.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:45:06 smithi016 ceph-mon[29101]: pgmap v1504: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:45:07.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:45:06 smithi016 ceph-mon[37215]: pgmap v1504: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:45:09.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:45:08 smithi183 ceph-mon[33745]: pgmap v1505: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:09.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:45:08 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:45:09.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:45:08 smithi016 ceph-mon[29101]: pgmap v1505: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:09.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:45:08 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:45:09.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:45:08 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:45:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:45:09.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:45:08 smithi016 ceph-mon[37215]: pgmap v1505: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:09.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:45:08 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:45:11.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:45:10 smithi183 ceph-mon[33745]: pgmap v1506: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:45:11.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:45:10 smithi016 ceph-mon[37215]: pgmap v1506: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:45:11.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:45:10 smithi016 ceph-mon[29101]: pgmap v1506: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:45:13.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:45:12 smithi183 ceph-mon[33745]: pgmap v1507: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:13.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:45:12 smithi016 ceph-mon[29101]: pgmap v1507: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:13.177 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:45:12 smithi016 ceph-mon[37215]: pgmap v1507: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:13.791 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:45:13 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:45:15.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:45:14 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:45:15.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:45:14 smithi183 ceph-mon[33745]: pgmap v1508: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:15.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:45:14 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:45:15.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:45:14 smithi016 ceph-mon[29101]: pgmap v1508: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:15.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:45:14 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:45:15.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:45:14 smithi016 ceph-mon[37215]: pgmap v1508: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:17.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:45:16 smithi183 ceph-mon[33745]: pgmap v1509: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:45:17.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:45:16 smithi016 ceph-mon[37215]: pgmap v1509: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:45:17.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:45:16 smithi016 ceph-mon[29101]: pgmap v1509: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:45:19.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:45:18 smithi183 ceph-mon[33745]: pgmap v1510: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:19.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:45:18 smithi016 ceph-mon[29101]: pgmap v1510: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:19.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:45:18 smithi016 ceph-mon[37215]: pgmap v1510: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:19.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:45:18 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:45:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:45:21.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:45:20 smithi183 ceph-mon[33745]: pgmap v1511: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:45:21.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:45:20 smithi016 ceph-mon[29101]: pgmap v1511: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:45:21.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:45:20 smithi016 ceph-mon[37215]: pgmap v1511: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:45:23.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:45:22 smithi016 ceph-mon[37215]: pgmap v1512: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:23.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:45:22 smithi016 ceph-mon[29101]: pgmap v1512: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:23.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:45:22 smithi183 ceph-mon[33745]: pgmap v1512: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:23.791 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:45:23 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:45:24.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:45:23 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:45:24.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:45:23 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:45:24.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:45:23 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:45:25.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:45:24 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:45:25.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:45:24 smithi016 ceph-mon[37215]: pgmap v1513: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:25.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:45:24 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:45:25.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:45:24 smithi016 ceph-mon[29101]: pgmap v1513: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:25.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:45:24 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:45:25.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:45:24 smithi183 ceph-mon[33745]: pgmap v1513: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:27.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:45:26 smithi016 ceph-mon[37215]: pgmap v1514: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:27.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:45:26 smithi016 ceph-mon[29101]: pgmap v1514: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:27.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:45:26 smithi183 ceph-mon[33745]: pgmap v1514: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:29.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:45:28 smithi016 ceph-mon[37215]: pgmap v1515: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:29.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:45:28 smithi016 ceph-mon[29101]: pgmap v1515: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:29.163 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:45:28 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:45:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:45:29.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:45:28 smithi183 ceph-mon[33745]: pgmap v1515: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:31.093 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:45:30 smithi183 ceph-mon[33745]: pgmap v1516: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:31.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:45:30 smithi016 ceph-mon[37215]: pgmap v1516: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:31.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:45:30 smithi016 ceph-mon[29101]: pgmap v1516: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:33.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:45:32 smithi016 ceph-mon[37215]: pgmap v1517: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:33.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:45:32 smithi016 ceph-mon[29101]: pgmap v1517: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:33.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:45:32 smithi183 ceph-mon[33745]: pgmap v1517: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:33.791 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:45:33 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:45:35.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:45:34 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:45:35.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:45:34 smithi016 ceph-mon[37215]: pgmap v1518: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:35.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:45:34 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:45:35.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:45:34 smithi016 ceph-mon[29101]: pgmap v1518: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:35.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:45:34 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:45:35.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:45:34 smithi183 ceph-mon[33745]: pgmap v1518: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:37.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:45:36 smithi016 ceph-mon[37215]: pgmap v1519: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:45:37.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:45:36 smithi016 ceph-mon[29101]: pgmap v1519: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:45:37.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:45:36 smithi183 ceph-mon[33745]: pgmap v1519: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:45:39.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:45:38 smithi016 ceph-mon[29101]: pgmap v1520: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:39.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:45:38 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:45:39.163 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:45:38 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:45:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:45:39.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:45:38 smithi016 ceph-mon[37215]: pgmap v1520: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:39.163 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:45:38 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:45:39.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:45:38 smithi183 ceph-mon[33745]: pgmap v1520: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:39.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:45:38 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:45:41.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:45:40 smithi016 ceph-mon[37215]: pgmap v1521: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:45:41.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:45:40 smithi016 ceph-mon[29101]: pgmap v1521: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:45:41.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:45:40 smithi183 ceph-mon[33745]: pgmap v1521: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:45:43.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:45:42 smithi016 ceph-mon[29101]: pgmap v1522: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:43.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:45:42 smithi016 ceph-mon[37215]: pgmap v1522: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:43.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:45:42 smithi183 ceph-mon[33745]: pgmap v1522: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:43.792 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:45:43 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:45:45.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:45:44 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:45:45.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:45:44 smithi016 ceph-mon[37215]: pgmap v1523: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:45.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:45:44 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:45:45.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:45:44 smithi016 ceph-mon[29101]: pgmap v1523: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:45.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:45:44 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:45:45.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:45:44 smithi183 ceph-mon[33745]: pgmap v1523: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:47.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:45:46 smithi016 ceph-mon[29101]: pgmap v1524: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:47.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:45:46 smithi016 ceph-mon[37215]: pgmap v1524: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:47.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:45:46 smithi183 ceph-mon[33745]: pgmap v1524: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:49.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:45:48 smithi016 ceph-mon[29101]: pgmap v1525: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:49.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:45:48 smithi016 ceph-mon[37215]: pgmap v1525: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:49.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:45:48 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:45:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:45:49.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:45:48 smithi183 ceph-mon[33745]: pgmap v1525: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:51.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:45:50 smithi016 ceph-mon[29101]: pgmap v1526: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:51.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:45:50 smithi016 ceph-mon[37215]: pgmap v1526: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:51.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:45:50 smithi183 ceph-mon[33745]: pgmap v1526: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:53.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:45:52 smithi016 ceph-mon[29101]: pgmap v1527: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:53.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:45:52 smithi016 ceph-mon[37215]: pgmap v1527: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:53.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:45:52 smithi183 ceph-mon[33745]: pgmap v1527: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:53.791 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:45:53 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:45:54.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:45:53 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:45:54.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:45:53 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:45:54.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:45:53 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:45:55.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:45:54 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:45:55.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:45:54 smithi016 ceph-mon[37215]: pgmap v1528: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:55.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:45:54 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:45:55.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:45:54 smithi016 ceph-mon[29101]: pgmap v1528: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:55.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:45:54 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:45:55.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:45:54 smithi183 ceph-mon[33745]: pgmap v1528: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:57.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:45:56 smithi016 ceph-mon[29101]: pgmap v1529: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:57.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:45:56 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:45:57.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:45:56 smithi016 ceph-mon[37215]: pgmap v1529: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:57.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:45:56 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:45:57.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:45:56 smithi183 ceph-mon[33745]: pgmap v1529: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:45:57.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:45:56 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:45:59.077 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:45:58 smithi016 ceph-mon[29101]: pgmap v1530: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:59.078 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:45:58 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:45:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:45:59.078 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:45:58 smithi016 ceph-mon[37215]: pgmap v1530: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:45:59.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:45:58 smithi183 ceph-mon[33745]: pgmap v1530: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:01.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:46:00 smithi016 ceph-mon[37215]: pgmap v1531: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:01.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:46:00 smithi016 ceph-mon[29101]: pgmap v1531: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:01.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:46:00 smithi183 ceph-mon[33745]: pgmap v1531: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:02.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:46:02 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:46:02.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:46:02 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:46:02.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:46:02 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:46:02.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:46:02 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:46:02.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:46:02 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:46:02.292 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:46:02 smithi183 ceph-mon[33745]: pgmap v1532: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:02.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:46:02 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:46:02.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:46:02 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:46:02.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:46:02 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:46:02.412 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:46:02 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:46:02.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:46:02 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:46:02.413 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:46:02 smithi016 ceph-mon[37215]: pgmap v1532: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:02.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:46:02 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:46:02.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:46:02 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:46:02.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:46:02 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:46:02.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:46:02 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:46:02.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:46:02 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:46:02.413 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:46:02 smithi016 ceph-mon[29101]: pgmap v1532: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:03.791 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:46:03 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:46:04.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:46:04 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:04.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:46:04 smithi016 ceph-mon[37215]: pgmap v1533: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:04.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:46:04 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:04.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:46:04 smithi016 ceph-mon[29101]: pgmap v1533: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:05.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:46:04 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:05.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:46:04 smithi183 ceph-mon[33745]: pgmap v1533: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:06.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:46:06 smithi016 ceph-mon[37215]: pgmap v1534: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:06.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:46:06 smithi016 ceph-mon[29101]: pgmap v1534: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:07.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:46:06 smithi183 ceph-mon[33745]: pgmap v1534: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:08.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:46:08 smithi016 ceph-mon[29101]: pgmap v1535: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:08.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:46:08 smithi016 ceph-mon[37215]: pgmap v1535: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:08.913 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:46:08 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:46:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:46:09.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:46:08 smithi183 ceph-mon[33745]: pgmap v1535: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:09.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:46:09 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:46:09.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:46:09 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:46:10.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:46:09 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:46:10.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:46:10 smithi016 ceph-mon[29101]: pgmap v1536: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:10.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:46:10 smithi016 ceph-mon[37215]: pgmap v1536: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:11.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:46:10 smithi183 ceph-mon[33745]: pgmap v1536: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:12.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:46:12 smithi016 ceph-mon[29101]: pgmap v1537: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:12.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:46:12 smithi016 ceph-mon[37215]: pgmap v1537: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:13.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:46:12 smithi183 ceph-mon[33745]: pgmap v1537: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:13.791 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:46:13 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:46:14.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:46:14 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:14.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:46:14 smithi016 ceph-mon[37215]: pgmap v1538: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:14.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:46:14 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:14.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:46:14 smithi016 ceph-mon[29101]: pgmap v1538: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:15.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:46:14 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:15.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:46:14 smithi183 ceph-mon[33745]: pgmap v1538: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:16.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:46:16 smithi016 ceph-mon[37215]: pgmap v1539: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:46:16.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:46:16 smithi016 ceph-mon[29101]: pgmap v1539: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:46:17.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:46:16 smithi183 ceph-mon[33745]: pgmap v1539: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:46:18.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:46:18 smithi016 ceph-mon[37215]: pgmap v1540: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:18.913 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:46:18 smithi016 ceph-mon[29101]: pgmap v1540: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:18.913 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:46:18 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:46:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:46:19.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:46:18 smithi183 ceph-mon[33745]: pgmap v1540: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:20.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:46:20 smithi016 ceph-mon[37215]: pgmap v1541: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:20.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:46:20 smithi016 ceph-mon[29101]: pgmap v1541: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:21.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:46:20 smithi183 ceph-mon[33745]: pgmap v1541: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:22.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:46:22 smithi016 ceph-mon[37215]: pgmap v1542: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:22.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:46:22 smithi016 ceph-mon[29101]: pgmap v1542: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:23.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:46:22 smithi183 ceph-mon[33745]: pgmap v1542: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:23.791 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:46:23 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:46:24.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:46:24 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:24.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:46:24 smithi016 ceph-mon[29101]: pgmap v1543: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:24.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:46:24 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:46:24.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:46:24 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:24.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:46:24 smithi016 ceph-mon[37215]: pgmap v1543: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:24.913 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:46:24 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:46:25.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:46:24 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:25.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:46:24 smithi183 ceph-mon[33745]: pgmap v1543: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:25.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:46:24 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:46:26.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:46:26 smithi016 ceph-mon[29101]: pgmap v1544: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:26.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:46:26 smithi016 ceph-mon[37215]: pgmap v1544: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:27.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:46:26 smithi183 ceph-mon[33745]: pgmap v1544: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:28.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:46:28 smithi016 ceph-mon[37215]: pgmap v1545: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:28.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:46:28 smithi016 ceph-mon[29101]: pgmap v1545: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:28.913 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:46:28 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:46:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:46:29.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:46:28 smithi183 ceph-mon[33745]: pgmap v1545: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:30.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:46:30 smithi016 ceph-mon[29101]: pgmap v1546: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:30.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:46:30 smithi016 ceph-mon[37215]: pgmap v1546: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:31.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:46:30 smithi183 ceph-mon[33745]: pgmap v1546: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:32.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:46:32 smithi016 ceph-mon[29101]: pgmap v1547: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:32.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:46:32 smithi016 ceph-mon[37215]: pgmap v1547: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:33.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:46:32 smithi183 ceph-mon[33745]: pgmap v1547: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:33.791 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:46:33 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:46:34.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:46:34 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:34.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:46:34 smithi016 ceph-mon[29101]: pgmap v1548: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:34.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:46:34 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:34.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:46:34 smithi016 ceph-mon[37215]: pgmap v1548: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:35.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:46:34 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:35.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:46:34 smithi183 ceph-mon[33745]: pgmap v1548: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:36.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:46:36 smithi016 ceph-mon[37215]: pgmap v1549: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:36.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:46:36 smithi016 ceph-mon[29101]: pgmap v1549: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:37.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:46:36 smithi183 ceph-mon[33745]: pgmap v1549: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:38.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:46:38 smithi016 ceph-mon[37215]: pgmap v1550: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:38.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:46:38 smithi016 ceph-mon[29101]: pgmap v1550: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:38.913 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:46:38 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:46:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:46:39.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:46:38 smithi183 ceph-mon[33745]: pgmap v1550: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:39.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:46:39 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:46:39.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:46:39 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:46:40.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:46:39 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:46:40.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:46:40 smithi016 ceph-mon[29101]: pgmap v1551: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:40.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:46:40 smithi016 ceph-mon[37215]: pgmap v1551: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:41.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:46:40 smithi183 ceph-mon[33745]: pgmap v1551: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:42.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:46:42 smithi016 ceph-mon[37215]: pgmap v1552: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:42.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:46:42 smithi016 ceph-mon[29101]: pgmap v1552: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:43.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:46:42 smithi183 ceph-mon[33745]: pgmap v1552: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:43.791 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:46:43 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:46:44.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:46:44 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:44.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:46:44 smithi016 ceph-mon[29101]: pgmap v1553: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:44.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:46:44 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:44.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:46:44 smithi016 ceph-mon[37215]: pgmap v1553: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:45.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:46:44 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:45.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:46:44 smithi183 ceph-mon[33745]: pgmap v1553: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:46.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:46:46 smithi016 ceph-mon[37215]: pgmap v1554: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:46:46.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:46:46 smithi016 ceph-mon[29101]: pgmap v1554: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:46:47.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:46:46 smithi183 ceph-mon[33745]: pgmap v1554: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:46:48.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:46:48 smithi016 ceph-mon[37215]: pgmap v1555: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:48.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:46:48 smithi016 ceph-mon[29101]: pgmap v1555: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:48.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:46:48 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:46:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:46:49.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:46:48 smithi183 ceph-mon[33745]: pgmap v1555: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:51.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:46:50 smithi183 ceph-mon[33745]: pgmap v1556: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:46:51.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:46:50 smithi016 ceph-mon[37215]: pgmap v1556: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:46:51.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:46:50 smithi016 ceph-mon[29101]: pgmap v1556: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:46:53.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:46:52 smithi183 ceph-mon[33745]: pgmap v1557: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:53.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:46:52 smithi016 ceph-mon[29101]: pgmap v1557: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:53.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:46:52 smithi016 ceph-mon[37215]: pgmap v1557: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:46:53.791 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:46:53 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:46:55.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:46:54 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:55.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:46:54 smithi183 ceph-mon[33745]: pgmap v1558: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:55.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:46:54 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:46:55.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:46:54 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:55.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:46:54 smithi016 ceph-mon[37215]: pgmap v1558: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:55.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:46:54 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:46:55.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:46:54 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:46:55.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:46:54 smithi016 ceph-mon[29101]: pgmap v1558: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:55.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:46:54 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:46:57.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:46:56 smithi183 ceph-mon[33745]: pgmap v1559: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:46:57.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:46:56 smithi016 ceph-mon[37215]: pgmap v1559: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:46:57.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:46:56 smithi016 ceph-mon[29101]: pgmap v1559: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:46:59.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:46:58 smithi183 ceph-mon[33745]: pgmap v1560: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:59.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:46:58 smithi016 ceph-mon[29101]: pgmap v1560: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:59.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:46:58 smithi016 ceph-mon[37215]: pgmap v1560: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:46:59.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:46:58 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:46:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:47:00.964 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:00 smithi016 ceph-mon[37215]: pgmap v1561: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:47:00.964 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:00 smithi016 ceph-mon[29101]: pgmap v1561: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:47:01.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:00 smithi183 ceph-mon[33745]: pgmap v1561: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:47:02.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:01 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:47:02.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:01 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:47:02.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:01 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:47:02.683 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:02 smithi183 ceph-mon[33745]: pgmap v1562: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:03.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:02 smithi016 ceph-mon[37215]: pgmap v1562: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:03.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:02 smithi016 ceph-mon[29101]: pgmap v1562: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:03.791 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:47:03 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:47:04.927 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:04 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:04.927 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:04 smithi183 ceph-mon[33745]: pgmap v1563: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:05.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:04 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:05.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:04 smithi016 ceph-mon[37215]: pgmap v1563: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:05.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:04 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:05.163 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:04 smithi016 ceph-mon[29101]: pgmap v1563: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:06.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:06 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:47:06.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:06 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:47:06.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:06 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:47:06.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:06 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:47:06.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:06 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:47:06.662 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:06 smithi016 ceph-mon[29101]: pgmap v1564: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:47:06.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:06 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:47:06.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:06 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:47:06.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:06 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:47:06.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:06 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:47:06.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:06 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:47:06.663 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:06 smithi016 ceph-mon[37215]: pgmap v1564: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:47:06.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:06 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:47:06.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:06 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:47:06.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:06 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:47:06.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:06 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:47:06.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:06 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:47:06.792 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:06 smithi183 ceph-mon[33745]: pgmap v1564: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:47:08.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:08 smithi016 ceph-mon[37215]: pgmap v1565: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:08.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:08 smithi016 ceph-mon[29101]: pgmap v1565: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:08.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:47:08 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:47:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:47:09.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:08 smithi183 ceph-mon[33745]: pgmap v1565: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:09.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:09 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:47:09.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:09 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:47:10.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:09 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:47:10.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:10 smithi016 ceph-mon[37215]: pgmap v1566: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:47:10.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:10 smithi016 ceph-mon[29101]: pgmap v1566: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:47:11.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:10 smithi183 ceph-mon[33745]: pgmap v1566: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:47:12.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:12 smithi016 ceph-mon[29101]: pgmap v1567: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:12.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:12 smithi016 ceph-mon[37215]: pgmap v1567: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:13.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:12 smithi183 ceph-mon[33745]: pgmap v1567: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:13.791 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:47:13 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:47:14.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:14 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:14.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:14 smithi016 ceph-mon[29101]: pgmap v1568: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:14.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:14 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:14.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:14 smithi016 ceph-mon[37215]: pgmap v1568: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:15.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:14 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:15.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:14 smithi183 ceph-mon[33745]: pgmap v1568: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:16.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:16 smithi016 ceph-mon[37215]: pgmap v1569: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:47:16.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:16 smithi016 ceph-mon[29101]: pgmap v1569: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:47:17.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:16 smithi183 ceph-mon[33745]: pgmap v1569: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:47:18.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:18 smithi016 ceph-mon[29101]: pgmap v1570: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:18.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:18 smithi016 ceph-mon[37215]: pgmap v1570: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:18.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:47:18 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:47:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:47:19.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:18 smithi183 ceph-mon[33745]: pgmap v1570: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:20.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:20 smithi016 ceph-mon[37215]: pgmap v1571: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:47:20.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:20 smithi016 ceph-mon[29101]: pgmap v1571: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:47:21.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:20 smithi183 ceph-mon[33745]: pgmap v1571: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:47:22.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:22 smithi016 ceph-mon[37215]: pgmap v1572: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:22.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:22 smithi016 ceph-mon[29101]: pgmap v1572: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:23.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:22 smithi183 ceph-mon[33745]: pgmap v1572: 228 pgs: 228 active+clean; 582 KiB data, 754 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:23.791 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:47:23 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:47:24.911 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:24 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:24.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:24 smithi016 ceph-mon[29101]: pgmap v1573: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T04:47:24.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:24 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:47:24.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:24 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:24.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:24 smithi016 ceph-mon[37215]: pgmap v1573: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T04:47:24.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:24 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:47:25.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:24 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:25.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:24 smithi183 ceph-mon[33745]: pgmap v1573: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T04:47:25.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:24 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:47:26.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:26 smithi016 ceph-mon[37215]: pgmap v1574: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T04:47:26.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:26 smithi016 ceph-mon[29101]: pgmap v1574: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T04:47:27.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:26 smithi183 ceph-mon[33745]: pgmap v1574: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T04:47:28.911 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:28 smithi016 ceph-mon[37215]: pgmap v1575: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T04:47:28.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:28 smithi016 ceph-mon[29101]: pgmap v1575: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T04:47:28.912 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:47:28 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:47:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:47:29.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:28 smithi183 ceph-mon[33745]: pgmap v1575: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T04:47:30.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:30 smithi016 ceph-mon[37215]: pgmap v1576: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T04:47:30.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:30 smithi016 ceph-mon[29101]: pgmap v1576: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T04:47:31.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:30 smithi183 ceph-mon[33745]: pgmap v1576: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T04:47:32.911 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:32 smithi016 ceph-mon[37215]: pgmap v1577: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T04:47:32.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:32 smithi016 ceph-mon[29101]: pgmap v1577: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T04:47:33.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:32 smithi183 ceph-mon[33745]: pgmap v1577: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T04:47:33.791 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:47:33 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:47:34.911 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:34 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:34.912 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:34 smithi016 ceph-mon[37215]: pgmap v1578: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T04:47:34.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:34 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:34.912 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:34 smithi016 ceph-mon[29101]: pgmap v1578: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T04:47:35.035 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:34 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:35.036 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:34 smithi183 ceph-mon[33745]: pgmap v1578: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 37 KiB/s rd, 0 B/s wr, 60 op/s 2024-03-26T04:47:37.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:36 smithi183 ceph-mon[33745]: pgmap v1579: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:47:37.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:36 smithi016 ceph-mon[37215]: pgmap v1579: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:47:37.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:36 smithi016 ceph-mon[29101]: pgmap v1579: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:47:39.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:38 smithi183 ceph-mon[33745]: pgmap v1580: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:39.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:38 smithi016 ceph-mon[29101]: pgmap v1580: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:39.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:38 smithi016 ceph-mon[37215]: pgmap v1580: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:39.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:47:38 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:47:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:47:40.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:39 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:47:40.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:39 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:47:40.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:39 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:47:41.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:40 smithi183 ceph-mon[33745]: pgmap v1581: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:41.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:40 smithi016 ceph-mon[37215]: pgmap v1581: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:41.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:40 smithi016 ceph-mon[29101]: pgmap v1581: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:43.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:42 smithi183 ceph-mon[33745]: pgmap v1582: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:43.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:42 smithi016 ceph-mon[29101]: pgmap v1582: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:43.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:42 smithi016 ceph-mon[37215]: pgmap v1582: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:43.791 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:47:43 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:47:44.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:44 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:44.791 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:44 smithi183 ceph-mon[33745]: pgmap v1583: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:45.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:44 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:45.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:44 smithi016 ceph-mon[29101]: pgmap v1583: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:45.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:44 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:45.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:44 smithi016 ceph-mon[37215]: pgmap v1583: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:47.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:46 smithi183 ceph-mon[33745]: pgmap v1584: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:47:47.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:46 smithi016 ceph-mon[37215]: pgmap v1584: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:47:47.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:46 smithi016 ceph-mon[29101]: pgmap v1584: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:47:49.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:48 smithi183 ceph-mon[33745]: pgmap v1585: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:49.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:48 smithi016 ceph-mon[29101]: pgmap v1585: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:49.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:48 smithi016 ceph-mon[37215]: pgmap v1585: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:49.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:47:48 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:47:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:47:51.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:50 smithi183 ceph-mon[33745]: pgmap v1586: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:47:51.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:50 smithi016 ceph-mon[29101]: pgmap v1586: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:47:51.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:50 smithi016 ceph-mon[37215]: pgmap v1586: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:47:53.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:52 smithi183 ceph-mon[33745]: pgmap v1587: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:53.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:52 smithi016 ceph-mon[37215]: pgmap v1587: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:53.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:52 smithi016 ceph-mon[29101]: pgmap v1587: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:47:53.790 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:47:53 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:47:55.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:54 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:55.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:54 smithi183 ceph-mon[33745]: pgmap v1588: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:55.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:54 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:47:55.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:54 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:55.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:54 smithi016 ceph-mon[37215]: pgmap v1588: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:55.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:54 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:47:55.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:54 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:47:55.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:54 smithi016 ceph-mon[29101]: pgmap v1588: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:55.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:54 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:47:57.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:56 smithi183 ceph-mon[33745]: pgmap v1589: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:47:57.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:56 smithi016 ceph-mon[29101]: pgmap v1589: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:47:57.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:56 smithi016 ceph-mon[37215]: pgmap v1589: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:47:59.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:47:58 smithi183 ceph-mon[33745]: pgmap v1590: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:59.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:47:58 smithi016 ceph-mon[29101]: pgmap v1590: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:59.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:47:58 smithi016 ceph-mon[37215]: pgmap v1590: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:47:59.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:47:58 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:47:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:48:00.963 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:00 smithi016 ceph-mon[29101]: pgmap v1591: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:48:00.964 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:00 smithi016 ceph-mon[37215]: pgmap v1591: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:48:01.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:00 smithi183 ceph-mon[33745]: pgmap v1591: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:48:03.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:02 smithi183 ceph-mon[33745]: pgmap v1592: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:48:03.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:02 smithi016 ceph-mon[29101]: pgmap v1592: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:48:03.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:02 smithi016 ceph-mon[37215]: pgmap v1592: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:48:03.790 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:48:03 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:48:05.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:04 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:05.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:04 smithi183 ceph-mon[33745]: pgmap v1593: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:48:05.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:04 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:05.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:04 smithi016 ceph-mon[29101]: pgmap v1593: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:48:05.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:04 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:05.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:04 smithi016 ceph-mon[37215]: pgmap v1593: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:48:06.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:05 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:48:06.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:05 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:48:06.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:05 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:48:07.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:06 smithi183 ceph-mon[33745]: pgmap v1594: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:48:07.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:06 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:48:07.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:06 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:48:07.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:06 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:48:07.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:06 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:48:07.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:06 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:48:07.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:06 smithi016 ceph-mon[29101]: pgmap v1594: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:48:07.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:06 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:48:07.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:06 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:48:07.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:06 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:48:07.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:06 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:48:07.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:06 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:48:07.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:06 smithi016 ceph-mon[37215]: pgmap v1594: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:48:07.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:06 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:48:07.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:06 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:48:07.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:06 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:48:07.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:06 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:48:07.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:06 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:48:09.041 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:08 smithi183 ceph-mon[33745]: pgmap v1595: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:48:09.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:08 smithi016 ceph-mon[29101]: pgmap v1595: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:48:09.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:08 smithi016 ceph-mon[37215]: pgmap v1595: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:48:09.161 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:48:08 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:48:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:48:10.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:09 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:48:10.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:09 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:48:10.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:09 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:48:11.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:10 smithi183 ceph-mon[33745]: pgmap v1596: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:48:11.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:10 smithi016 ceph-mon[37215]: pgmap v1596: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:48:11.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:10 smithi016 ceph-mon[29101]: pgmap v1596: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:48:13.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:12 smithi183 ceph-mon[33745]: pgmap v1597: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:48:13.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:12 smithi016 ceph-mon[29101]: pgmap v1597: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:48:13.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:12 smithi016 ceph-mon[37215]: pgmap v1597: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:48:14.040 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:48:13 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:48:15.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:14 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:15.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:14 smithi183 ceph-mon[33745]: pgmap v1598: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:48:15.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:14 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:15.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:14 smithi016 ceph-mon[37215]: pgmap v1598: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:48:15.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:14 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:15.162 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:14 smithi016 ceph-mon[29101]: pgmap v1598: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:48:17.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:16 smithi183 ceph-mon[33745]: pgmap v1599: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:48:17.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:16 smithi016 ceph-mon[29101]: pgmap v1599: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:48:17.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:16 smithi016 ceph-mon[37215]: pgmap v1599: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:48:19.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:18 smithi183 ceph-mon[33745]: pgmap v1600: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:48:19.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:18 smithi016 ceph-mon[37215]: pgmap v1600: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:48:19.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:18 smithi016 ceph-mon[29101]: pgmap v1600: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:48:19.162 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:48:18 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:48:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:48:21.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:20 smithi183 ceph-mon[33745]: pgmap v1601: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:48:21.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:20 smithi016 ceph-mon[29101]: pgmap v1601: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:48:21.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:20 smithi016 ceph-mon[37215]: pgmap v1601: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:48:23.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:22 smithi183 ceph-mon[33745]: pgmap v1602: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:48:23.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:22 smithi016 ceph-mon[29101]: pgmap v1602: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:48:23.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:22 smithi016 ceph-mon[37215]: pgmap v1602: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:48:24.040 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:48:23 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:48:25.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:24 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:25.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:24 smithi183 ceph-mon[33745]: pgmap v1603: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:48:25.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:24 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:48:25.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:24 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:25.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:24 smithi016 ceph-mon[29101]: pgmap v1603: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:48:25.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:24 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:48:25.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:24 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:25.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:24 smithi016 ceph-mon[37215]: pgmap v1603: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:48:25.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:24 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:48:27.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:26 smithi183 ceph-mon[33745]: pgmap v1604: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:48:27.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:26 smithi016 ceph-mon[37215]: pgmap v1604: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:48:27.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:26 smithi016 ceph-mon[29101]: pgmap v1604: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:48:29.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:28 smithi183 ceph-mon[33745]: pgmap v1605: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:48:29.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:28 smithi016 ceph-mon[37215]: pgmap v1605: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:48:29.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:28 smithi016 ceph-mon[29101]: pgmap v1605: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:48:29.161 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:48:28 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:48:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:48:31.042 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:30 smithi183 ceph-mon[33745]: pgmap v1606: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:48:31.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:30 smithi016 ceph-mon[37215]: pgmap v1606: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:48:31.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:30 smithi016 ceph-mon[29101]: pgmap v1606: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:48:33.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:32 smithi183 ceph-mon[33745]: pgmap v1607: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:48:33.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:32 smithi016 ceph-mon[37215]: pgmap v1607: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:48:33.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:32 smithi016 ceph-mon[29101]: pgmap v1607: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:48:34.040 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:48:33 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:48:35.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:34 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:35.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:34 smithi183 ceph-mon[33745]: pgmap v1608: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:48:35.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:34 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:35.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:34 smithi016 ceph-mon[37215]: pgmap v1608: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:48:35.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:34 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:35.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:34 smithi016 ceph-mon[29101]: pgmap v1608: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:48:37.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:36 smithi183 ceph-mon[33745]: pgmap v1609: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:48:37.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:36 smithi016 ceph-mon[29101]: pgmap v1609: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:48:37.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:36 smithi016 ceph-mon[37215]: pgmap v1609: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:48:39.043 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:38 smithi183 ceph-mon[33745]: pgmap v1610: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:48:39.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:38 smithi016 ceph-mon[29101]: pgmap v1610: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:48:39.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:38 smithi016 ceph-mon[37215]: pgmap v1610: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:48:39.161 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:48:38 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:48:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:48:40.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:39 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:48:40.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:39 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:48:40.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:39 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:48:41.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:40 smithi183 ceph-mon[33745]: pgmap v1611: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:48:41.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:40 smithi016 ceph-mon[29101]: pgmap v1611: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:48:41.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:40 smithi016 ceph-mon[37215]: pgmap v1611: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:48:43.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:42 smithi183 ceph-mon[33745]: pgmap v1612: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:48:43.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:42 smithi016 ceph-mon[37215]: pgmap v1612: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:48:43.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:42 smithi016 ceph-mon[29101]: pgmap v1612: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:48:44.040 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:48:43 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:48:45.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:44 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:45.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:44 smithi183 ceph-mon[33745]: pgmap v1613: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:48:45.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:44 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:45.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:44 smithi016 ceph-mon[37215]: pgmap v1613: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:48:45.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:44 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:45.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:44 smithi016 ceph-mon[29101]: pgmap v1613: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:48:47.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:46 smithi183 ceph-mon[33745]: pgmap v1614: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:48:47.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:46 smithi016 ceph-mon[37215]: pgmap v1614: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:48:47.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:46 smithi016 ceph-mon[29101]: pgmap v1614: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:48:49.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:48 smithi183 ceph-mon[33745]: pgmap v1615: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:48:49.161 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:48:48 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:48:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:48:49.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:48 smithi016 ceph-mon[37215]: pgmap v1615: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:48:49.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:48 smithi016 ceph-mon[29101]: pgmap v1615: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:48:51.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:50 smithi183 ceph-mon[33745]: pgmap v1616: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:48:51.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:50 smithi016 ceph-mon[29101]: pgmap v1616: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:48:51.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:50 smithi016 ceph-mon[37215]: pgmap v1616: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:48:52.040 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:48:51 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a[63907]: logger=cleanup t=2024-03-26T04:48:51.765295453Z level=info msg="Completed cleanup jobs" duration=2.235934ms 2024-03-26T04:48:53.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:52 smithi183 ceph-mon[33745]: pgmap v1617: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:48:53.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:52 smithi016 ceph-mon[37215]: pgmap v1617: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:48:53.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:52 smithi016 ceph-mon[29101]: pgmap v1617: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:48:54.040 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:48:53 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:48:55.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:54 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:55.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:54 smithi183 ceph-mon[33745]: pgmap v1618: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:48:55.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:54 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:48:55.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:54 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:55.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:54 smithi016 ceph-mon[37215]: pgmap v1618: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:48:55.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:54 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:48:55.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:54 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:48:55.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:54 smithi016 ceph-mon[29101]: pgmap v1618: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:48:55.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:54 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:48:57.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:56 smithi183 ceph-mon[33745]: pgmap v1619: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:48:57.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:56 smithi016 ceph-mon[37215]: pgmap v1619: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:48:57.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:56 smithi016 ceph-mon[29101]: pgmap v1619: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:48:59.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:48:58 smithi016 ceph-mon[37215]: pgmap v1620: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:48:59.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:48:58 smithi016 ceph-mon[29101]: pgmap v1620: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:48:59.161 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:48:58 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:48:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:48:59.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:48:58 smithi183 ceph-mon[33745]: pgmap v1620: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:01.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:00 smithi016 ceph-mon[37215]: pgmap v1621: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:49:01.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:00 smithi016 ceph-mon[29101]: pgmap v1621: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:49:01.290 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:00 smithi183 ceph-mon[33745]: pgmap v1621: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:49:03.092 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:02 smithi183 ceph-mon[33745]: pgmap v1622: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:03.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:02 smithi016 ceph-mon[29101]: pgmap v1622: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:03.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:02 smithi016 ceph-mon[37215]: pgmap v1622: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:04.039 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:49:03 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:49:05.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:04 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:05.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:04 smithi016 ceph-mon[37215]: pgmap v1623: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:05.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:04 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:05.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:04 smithi016 ceph-mon[29101]: pgmap v1623: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:05.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:04 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:05.290 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:04 smithi183 ceph-mon[33745]: pgmap v1623: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:07.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:06 smithi016 ceph-mon[37215]: pgmap v1624: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:49:07.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:06 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:49:07.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:06 smithi016 ceph-mon[29101]: pgmap v1624: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:49:07.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:06 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:49:07.290 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:06 smithi183 ceph-mon[33745]: pgmap v1624: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:49:07.290 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:06 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:49:09.039 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:08 smithi183 ceph-mon[33745]: pgmap v1625: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:09.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:08 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:49:09.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:08 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:49:09.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:08 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:49:09.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:08 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:49:09.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:08 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:49:09.160 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:49:08 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:49:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:49:09.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:08 smithi016 ceph-mon[29101]: pgmap v1625: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:09.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:08 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:49:09.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:08 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:49:09.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:08 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:49:09.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:08 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:49:09.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:08 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:49:09.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:08 smithi016 ceph-mon[37215]: pgmap v1625: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:09.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:08 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:49:09.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:08 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:49:09.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:08 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:49:09.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:08 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:49:09.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:08 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:49:10.039 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:09 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:49:10.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:09 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:49:10.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:09 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:49:11.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:10 smithi183 ceph-mon[33745]: pgmap v1626: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:49:11.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:10 smithi016 ceph-mon[37215]: pgmap v1626: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:49:11.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:10 smithi016 ceph-mon[29101]: pgmap v1626: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:49:13.039 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:12 smithi183 ceph-mon[33745]: pgmap v1627: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:13.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:12 smithi016 ceph-mon[37215]: pgmap v1627: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:13.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:12 smithi016 ceph-mon[29101]: pgmap v1627: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:14.039 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:49:13 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:49:15.039 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:14 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:15.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:14 smithi183 ceph-mon[33745]: pgmap v1628: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:15.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:14 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:15.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:14 smithi016 ceph-mon[37215]: pgmap v1628: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:15.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:14 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:15.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:14 smithi016 ceph-mon[29101]: pgmap v1628: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:17.039 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:16 smithi183 ceph-mon[33745]: pgmap v1629: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:49:17.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:16 smithi016 ceph-mon[29101]: pgmap v1629: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:49:17.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:16 smithi016 ceph-mon[37215]: pgmap v1629: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:49:19.039 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:18 smithi183 ceph-mon[33745]: pgmap v1630: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:19.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:18 smithi016 ceph-mon[29101]: pgmap v1630: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:19.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:18 smithi016 ceph-mon[37215]: pgmap v1630: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:19.161 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:49:18 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:49:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:49:21.039 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:20 smithi183 ceph-mon[33745]: pgmap v1631: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:49:21.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:20 smithi016 ceph-mon[29101]: pgmap v1631: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:49:21.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:20 smithi016 ceph-mon[37215]: pgmap v1631: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:49:23.039 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:22 smithi183 ceph-mon[33745]: pgmap v1632: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:23.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:22 smithi016 ceph-mon[29101]: pgmap v1632: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:23.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:22 smithi016 ceph-mon[37215]: pgmap v1632: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:24.039 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:49:23 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:49:25.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:24 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:25.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:24 smithi183 ceph-mon[33745]: pgmap v1633: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:25.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:24 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:49:25.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:24 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:25.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:24 smithi016 ceph-mon[29101]: pgmap v1633: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:25.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:24 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:49:25.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:24 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:25.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:24 smithi016 ceph-mon[37215]: pgmap v1633: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:25.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:24 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:49:27.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:26 smithi183 ceph-mon[33745]: pgmap v1634: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:49:27.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:26 smithi016 ceph-mon[29101]: pgmap v1634: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:49:27.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:26 smithi016 ceph-mon[37215]: pgmap v1634: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:49:29.039 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:28 smithi183 ceph-mon[33745]: pgmap v1635: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:29.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:28 smithi016 ceph-mon[29101]: pgmap v1635: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:29.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:28 smithi016 ceph-mon[37215]: pgmap v1635: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:29.161 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:49:28 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:49:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:49:31.039 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:30 smithi183 ceph-mon[33745]: pgmap v1636: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:49:31.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:30 smithi016 ceph-mon[29101]: pgmap v1636: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:49:31.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:30 smithi016 ceph-mon[37215]: pgmap v1636: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:49:33.039 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:32 smithi183 ceph-mon[33745]: pgmap v1637: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:33.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:32 smithi016 ceph-mon[29101]: pgmap v1637: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:33.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:32 smithi016 ceph-mon[37215]: pgmap v1637: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:33.941 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:49:33 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:49:35.039 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:34 smithi183 ceph-mon[33745]: pgmap v1638: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:35.039 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:34 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:35.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:34 smithi016 ceph-mon[29101]: pgmap v1638: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:35.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:34 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:35.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:34 smithi016 ceph-mon[37215]: pgmap v1638: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:35.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:34 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:37.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:36 smithi016 ceph-mon[29101]: pgmap v1639: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:49:37.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:36 smithi016 ceph-mon[37215]: pgmap v1639: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:49:37.290 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:36 smithi183 ceph-mon[33745]: pgmap v1639: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:49:39.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:38 smithi016 ceph-mon[37215]: pgmap v1640: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:39.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:38 smithi016 ceph-mon[29101]: pgmap v1640: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:39.161 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:49:38 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:49:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:49:39.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:38 smithi183 ceph-mon[33745]: pgmap v1640: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:40.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:39 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:49:40.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:39 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:49:40.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:39 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:49:41.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:40 smithi016 ceph-mon[37215]: pgmap v1641: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:49:41.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:40 smithi016 ceph-mon[29101]: pgmap v1641: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:49:41.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:40 smithi183 ceph-mon[33745]: pgmap v1641: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:49:43.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:42 smithi016 ceph-mon[37215]: pgmap v1642: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:43.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:42 smithi016 ceph-mon[29101]: pgmap v1642: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:43.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:42 smithi183 ceph-mon[33745]: pgmap v1642: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:44.039 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:49:43 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:49:45.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:44 smithi016 ceph-mon[37215]: pgmap v1643: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:45.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:44 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:45.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:44 smithi016 ceph-mon[29101]: pgmap v1643: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:45.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:44 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:45.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:44 smithi183 ceph-mon[33745]: pgmap v1643: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:45.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:44 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:47.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:46 smithi016 ceph-mon[37215]: pgmap v1644: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:49:47.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:46 smithi016 ceph-mon[29101]: pgmap v1644: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:49:47.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:46 smithi183 ceph-mon[33745]: pgmap v1644: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:49:49.160 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:49:48 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:49:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:49:49.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:48 smithi016 ceph-mon[37215]: pgmap v1645: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:49.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:48 smithi016 ceph-mon[29101]: pgmap v1645: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:49.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:48 smithi183 ceph-mon[33745]: pgmap v1645: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:51.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:50 smithi016 ceph-mon[37215]: pgmap v1646: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:49:51.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:50 smithi016 ceph-mon[29101]: pgmap v1646: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:49:51.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:50 smithi183 ceph-mon[33745]: pgmap v1646: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:49:53.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:52 smithi016 ceph-mon[37215]: pgmap v1647: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:53.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:52 smithi016 ceph-mon[29101]: pgmap v1647: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:53.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:52 smithi183 ceph-mon[33745]: pgmap v1647: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:49:54.039 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:49:53 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:49:55.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:54 smithi016 ceph-mon[37215]: pgmap v1648: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:55.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:54 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:55.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:54 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:49:55.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:54 smithi016 ceph-mon[29101]: pgmap v1648: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:55.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:54 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:55.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:54 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:49:55.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:54 smithi183 ceph-mon[33745]: pgmap v1648: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:55.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:54 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:49:55.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:54 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:49:57.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:56 smithi016 ceph-mon[37215]: pgmap v1649: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:49:57.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:56 smithi016 ceph-mon[29101]: pgmap v1649: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:49:57.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:56 smithi183 ceph-mon[33745]: pgmap v1649: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:49:59.160 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:49:58 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:49:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:49:59.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:58 smithi016 ceph-mon[29101]: pgmap v1650: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:59.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:58 smithi016 ceph-mon[29101]: osdmap e755: 8 total, 8 up, 8 in 2024-03-26T04:49:59.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:58 smithi016 ceph-mon[37215]: pgmap v1650: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:59.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:58 smithi016 ceph-mon[37215]: osdmap e755: 8 total, 8 up, 8 in 2024-03-26T04:49:59.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:58 smithi183 ceph-mon[33745]: pgmap v1650: 228 pgs: 228 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:49:59.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:58 smithi183 ceph-mon[33745]: osdmap e755: 8 total, 8 up, 8 in 2024-03-26T04:50:00.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:59 smithi016 ceph-mon[29101]: osdmap e756: 8 total, 8 up, 8 in 2024-03-26T04:50:00.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:49:59 smithi016 ceph-mon[29101]: osdmap e757: 8 total, 8 up, 8 in 2024-03-26T04:50:00.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:59 smithi016 ceph-mon[37215]: osdmap e756: 8 total, 8 up, 8 in 2024-03-26T04:50:00.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:49:59 smithi016 ceph-mon[37215]: osdmap e757: 8 total, 8 up, 8 in 2024-03-26T04:50:00.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:59 smithi183 ceph-mon[33745]: osdmap e756: 8 total, 8 up, 8 in 2024-03-26T04:50:00.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:49:59 smithi183 ceph-mon[33745]: osdmap e757: 8 total, 8 up, 8 in 2024-03-26T04:50:01.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:00 smithi016 ceph-mon[37215]: pgmap v1653: 228 pgs: 64 unknown, 164 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:01.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:00 smithi016 ceph-mon[37215]: overall HEALTH_WARN 3 pool(s) do not have an application enabled 2024-03-26T04:50:01.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:00 smithi016 ceph-mon[29101]: pgmap v1653: 228 pgs: 64 unknown, 164 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:01.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:00 smithi016 ceph-mon[29101]: overall HEALTH_WARN 3 pool(s) do not have an application enabled 2024-03-26T04:50:01.290 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:00 smithi183 ceph-mon[33745]: pgmap v1653: 228 pgs: 64 unknown, 164 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:01.290 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:00 smithi183 ceph-mon[33745]: overall HEALTH_WARN 3 pool(s) do not have an application enabled 2024-03-26T04:50:02.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:01 smithi016 ceph-mon[37215]: osdmap e758: 8 total, 8 up, 8 in 2024-03-26T04:50:02.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:01 smithi016 ceph-mon[29101]: osdmap e758: 8 total, 8 up, 8 in 2024-03-26T04:50:02.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:01 smithi183 ceph-mon[33745]: osdmap e758: 8 total, 8 up, 8 in 2024-03-26T04:50:03.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:02 smithi016 ceph-mon[37215]: pgmap v1656: 164 pgs: 164 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:50:03.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:02 smithi016 ceph-mon[37215]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:03.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:02 smithi016 ceph-mon[37215]: osdmap e759: 8 total, 8 up, 8 in 2024-03-26T04:50:03.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:02 smithi016 ceph-mon[29101]: pgmap v1656: 164 pgs: 164 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:50:03.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:02 smithi016 ceph-mon[29101]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:03.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:02 smithi016 ceph-mon[29101]: osdmap e759: 8 total, 8 up, 8 in 2024-03-26T04:50:03.190 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:02 smithi183 ceph-mon[33745]: pgmap v1656: 164 pgs: 164 active+clean; 582 KiB data, 755 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:50:03.190 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:02 smithi183 ceph-mon[33745]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:03.190 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:02 smithi183 ceph-mon[33745]: osdmap e759: 8 total, 8 up, 8 in 2024-03-26T04:50:03.886 INFO:tasks.workunit.client.0.smithi016.stdout: list: Running main() from gmock_main.cc 2024-03-26T04:50:03.886 INFO:tasks.workunit.client.0.smithi016.stdout: list: [==========] Running 3 tests from 1 test suite. 2024-03-26T04:50:03.886 INFO:tasks.workunit.client.0.smithi016.stdout: list: [----------] Global test environment set-up. 2024-03-26T04:50:03.886 INFO:tasks.workunit.client.0.smithi016.stdout: list: [----------] 3 tests from NeoradosList 2024-03-26T04:50:03.886 INFO:tasks.workunit.client.0.smithi016.stdout: list: [ RUN ] NeoradosList.ListObjects 2024-03-26T04:50:03.886 INFO:tasks.workunit.client.0.smithi016.stdout: list: [ OK ] NeoradosList.ListObjects (1808894 ms) 2024-03-26T04:50:03.886 INFO:tasks.workunit.client.0.smithi016.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2024-03-26T04:50:03.886 INFO:tasks.workunit.client.0.smithi016.stdout: list: [ OK ] NeoradosList.ListObjectsNS (3031 ms) 2024-03-26T04:50:03.886 INFO:tasks.workunit.client.0.smithi016.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2024-03-26T04:50:03.887 INFO:tasks.workunit.client.0.smithi016.stdout: list: [ OK ] NeoradosList.ListObjectsMany (3021 ms) 2024-03-26T04:50:03.887 INFO:tasks.workunit.client.0.smithi016.stdout: list: [----------] 3 tests from NeoradosList (1814946 ms total) 2024-03-26T04:50:03.887 INFO:tasks.workunit.client.0.smithi016.stdout: list: 2024-03-26T04:50:03.887 INFO:tasks.workunit.client.0.smithi016.stdout: list: [----------] Global test environment tear-down 2024-03-26T04:50:03.887 INFO:tasks.workunit.client.0.smithi016.stdout: list: [==========] 3 tests from 1 test suite ran. (1814946 ms total) 2024-03-26T04:50:03.887 INFO:tasks.workunit.client.0.smithi016.stdout: list: [ PASSED ] 3 tests. 2024-03-26T04:50:04.039 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:03 smithi183 ceph-mon[33745]: osdmap e760: 8 total, 8 up, 8 in 2024-03-26T04:50:04.039 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:50:03 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:50:04.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:03 smithi016 ceph-mon[37215]: osdmap e760: 8 total, 8 up, 8 in 2024-03-26T04:50:04.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:03 smithi016 ceph-mon[29101]: osdmap e760: 8 total, 8 up, 8 in 2024-03-26T04:50:05.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:04 smithi016 ceph-mon[37215]: pgmap v1659: 228 pgs: 64 creating+peering, 164 active+clean; 582 KiB data, 793 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:05.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:04 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:05.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:04 smithi016 ceph-mon[37215]: osdmap e761: 8 total, 8 up, 8 in 2024-03-26T04:50:05.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:04 smithi016 ceph-mon[29101]: pgmap v1659: 228 pgs: 64 creating+peering, 164 active+clean; 582 KiB data, 793 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:05.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:04 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:05.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:04 smithi016 ceph-mon[29101]: osdmap e761: 8 total, 8 up, 8 in 2024-03-26T04:50:05.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:04 smithi183 ceph-mon[33745]: pgmap v1659: 228 pgs: 64 creating+peering, 164 active+clean; 582 KiB data, 793 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:05.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:04 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:05.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:04 smithi183 ceph-mon[33745]: osdmap e761: 8 total, 8 up, 8 in 2024-03-26T04:50:06.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:05 smithi016 ceph-mon[29101]: osdmap e762: 8 total, 8 up, 8 in 2024-03-26T04:50:06.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:05 smithi016 ceph-mon[37215]: osdmap e762: 8 total, 8 up, 8 in 2024-03-26T04:50:06.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:05 smithi183 ceph-mon[33745]: osdmap e762: 8 total, 8 up, 8 in 2024-03-26T04:50:07.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:06 smithi016 ceph-mon[37215]: pgmap v1662: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 793 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:07.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:06 smithi016 ceph-mon[37215]: osdmap e763: 8 total, 8 up, 8 in 2024-03-26T04:50:07.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:06 smithi016 ceph-mon[29101]: pgmap v1662: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 793 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:07.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:06 smithi016 ceph-mon[29101]: osdmap e763: 8 total, 8 up, 8 in 2024-03-26T04:50:07.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:06 smithi183 ceph-mon[33745]: pgmap v1662: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 793 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:07.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:06 smithi183 ceph-mon[33745]: osdmap e763: 8 total, 8 up, 8 in 2024-03-26T04:50:08.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:07 smithi016 ceph-mon[29101]: osdmap e764: 8 total, 8 up, 8 in 2024-03-26T04:50:08.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:07 smithi016 ceph-mon[29101]: pgmap v1665: 164 pgs: 164 active+clean; 582 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:08.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:07 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:50:08.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:07 smithi016 ceph-mon[37215]: osdmap e764: 8 total, 8 up, 8 in 2024-03-26T04:50:08.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:07 smithi016 ceph-mon[37215]: pgmap v1665: 164 pgs: 164 active+clean; 582 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:08.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:07 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:50:08.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:07 smithi183 ceph-mon[33745]: osdmap e764: 8 total, 8 up, 8 in 2024-03-26T04:50:08.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:07 smithi183 ceph-mon[33745]: pgmap v1665: 164 pgs: 164 active+clean; 582 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:08.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:07 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:50:09.160 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:50:08 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:50:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:50:09.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:08 smithi016 ceph-mon[29101]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:09.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:08 smithi016 ceph-mon[29101]: osdmap e765: 8 total, 8 up, 8 in 2024-03-26T04:50:09.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:08 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:50:09.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:08 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:50:09.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:08 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:50:09.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:08 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:50:09.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:08 smithi016 ceph-mon[37215]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:09.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:08 smithi016 ceph-mon[37215]: osdmap e765: 8 total, 8 up, 8 in 2024-03-26T04:50:09.161 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:08 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:50:09.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:08 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:50:09.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:08 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:50:09.162 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:08 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:50:09.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:08 smithi183 ceph-mon[33745]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:09.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:08 smithi183 ceph-mon[33745]: osdmap e765: 8 total, 8 up, 8 in 2024-03-26T04:50:09.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:08 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:50:09.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:08 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:50:09.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:08 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:50:09.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:08 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:50:10.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:09 smithi183 ceph-mon[33745]: osdmap e766: 8 total, 8 up, 8 in 2024-03-26T04:50:10.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:09 smithi183 ceph-mon[33745]: pgmap v1668: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:10.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:09 smithi016 ceph-mon[29101]: osdmap e766: 8 total, 8 up, 8 in 2024-03-26T04:50:10.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:09 smithi016 ceph-mon[29101]: pgmap v1668: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:10.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:09 smithi016 ceph-mon[37215]: osdmap e766: 8 total, 8 up, 8 in 2024-03-26T04:50:10.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:09 smithi016 ceph-mon[37215]: pgmap v1668: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 783 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:11.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:10 smithi183 ceph-mon[33745]: osdmap e767: 8 total, 8 up, 8 in 2024-03-26T04:50:11.409 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:10 smithi016 ceph-mon[37215]: osdmap e767: 8 total, 8 up, 8 in 2024-03-26T04:50:11.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:10 smithi016 ceph-mon[29101]: osdmap e767: 8 total, 8 up, 8 in 2024-03-26T04:50:12.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:11 smithi183 ceph-mon[33745]: osdmap e768: 8 total, 8 up, 8 in 2024-03-26T04:50:12.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:11 smithi183 ceph-mon[33745]: pgmap v1671: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 783 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:50:12.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:11 smithi016 ceph-mon[29101]: osdmap e768: 8 total, 8 up, 8 in 2024-03-26T04:50:12.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:11 smithi016 ceph-mon[29101]: pgmap v1671: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 783 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:50:12.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:11 smithi016 ceph-mon[37215]: osdmap e768: 8 total, 8 up, 8 in 2024-03-26T04:50:12.411 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:11 smithi016 ceph-mon[37215]: pgmap v1671: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 783 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:50:13.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:12 smithi183 ceph-mon[33745]: osdmap e769: 8 total, 8 up, 8 in 2024-03-26T04:50:13.409 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:12 smithi016 ceph-mon[37215]: osdmap e769: 8 total, 8 up, 8 in 2024-03-26T04:50:13.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:12 smithi016 ceph-mon[29101]: osdmap e769: 8 total, 8 up, 8 in 2024-03-26T04:50:13.942 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:50:13 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:50:14.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:13 smithi183 ceph-mon[33745]: osdmap e770: 8 total, 8 up, 8 in 2024-03-26T04:50:14.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:13 smithi183 ceph-mon[33745]: pgmap v1674: 164 pgs: 164 active+clean; 582 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:14.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:13 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:14.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:13 smithi016 ceph-mon[29101]: osdmap e770: 8 total, 8 up, 8 in 2024-03-26T04:50:14.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:13 smithi016 ceph-mon[29101]: pgmap v1674: 164 pgs: 164 active+clean; 582 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:14.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:13 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:14.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:13 smithi016 ceph-mon[37215]: osdmap e770: 8 total, 8 up, 8 in 2024-03-26T04:50:14.411 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:13 smithi016 ceph-mon[37215]: pgmap v1674: 164 pgs: 164 active+clean; 582 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:14.411 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:13 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:15.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:14 smithi183 ceph-mon[33745]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:15.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:14 smithi183 ceph-mon[33745]: osdmap e771: 8 total, 8 up, 8 in 2024-03-26T04:50:15.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:14 smithi016 ceph-mon[29101]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:15.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:14 smithi016 ceph-mon[29101]: osdmap e771: 8 total, 8 up, 8 in 2024-03-26T04:50:15.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:14 smithi016 ceph-mon[37215]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:15.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:14 smithi016 ceph-mon[37215]: osdmap e771: 8 total, 8 up, 8 in 2024-03-26T04:50:16.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:15 smithi183 ceph-mon[33745]: osdmap e772: 8 total, 8 up, 8 in 2024-03-26T04:50:16.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:15 smithi183 ceph-mon[33745]: pgmap v1677: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:16.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:15 smithi016 ceph-mon[29101]: osdmap e772: 8 total, 8 up, 8 in 2024-03-26T04:50:16.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:15 smithi016 ceph-mon[29101]: pgmap v1677: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:16.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:15 smithi016 ceph-mon[37215]: osdmap e772: 8 total, 8 up, 8 in 2024-03-26T04:50:16.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:15 smithi016 ceph-mon[37215]: pgmap v1677: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 792 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:17.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:16 smithi183 ceph-mon[33745]: osdmap e773: 8 total, 8 up, 8 in 2024-03-26T04:50:17.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:16 smithi016 ceph-mon[29101]: osdmap e773: 8 total, 8 up, 8 in 2024-03-26T04:50:17.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:16 smithi016 ceph-mon[37215]: osdmap e773: 8 total, 8 up, 8 in 2024-03-26T04:50:18.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:17 smithi183 ceph-mon[33745]: osdmap e774: 8 total, 8 up, 8 in 2024-03-26T04:50:18.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:17 smithi183 ceph-mon[33745]: pgmap v1680: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 825 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:18.409 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:17 smithi016 ceph-mon[37215]: osdmap e774: 8 total, 8 up, 8 in 2024-03-26T04:50:18.409 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:17 smithi016 ceph-mon[37215]: pgmap v1680: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 825 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:18.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:17 smithi016 ceph-mon[29101]: osdmap e774: 8 total, 8 up, 8 in 2024-03-26T04:50:18.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:17 smithi016 ceph-mon[29101]: pgmap v1680: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 825 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:19.159 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:50:18 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:50:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:50:19.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:18 smithi016 ceph-mon[37215]: osdmap e775: 8 total, 8 up, 8 in 2024-03-26T04:50:19.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:18 smithi016 ceph-mon[29101]: osdmap e775: 8 total, 8 up, 8 in 2024-03-26T04:50:19.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:18 smithi183 ceph-mon[33745]: osdmap e775: 8 total, 8 up, 8 in 2024-03-26T04:50:20.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:19 smithi183 ceph-mon[33745]: osdmap e776: 8 total, 8 up, 8 in 2024-03-26T04:50:20.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:19 smithi183 ceph-mon[33745]: pgmap v1683: 164 pgs: 164 active+clean; 582 KiB data, 825 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:20.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:19 smithi016 ceph-mon[29101]: osdmap e776: 8 total, 8 up, 8 in 2024-03-26T04:50:20.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:19 smithi016 ceph-mon[29101]: pgmap v1683: 164 pgs: 164 active+clean; 582 KiB data, 825 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:20.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:19 smithi016 ceph-mon[37215]: osdmap e776: 8 total, 8 up, 8 in 2024-03-26T04:50:20.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:19 smithi016 ceph-mon[37215]: pgmap v1683: 164 pgs: 164 active+clean; 582 KiB data, 825 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:21.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:20 smithi183 ceph-mon[33745]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:21.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:20 smithi183 ceph-mon[33745]: osdmap e777: 8 total, 8 up, 8 in 2024-03-26T04:50:21.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:20 smithi016 ceph-mon[29101]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:21.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:20 smithi016 ceph-mon[29101]: osdmap e777: 8 total, 8 up, 8 in 2024-03-26T04:50:21.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:20 smithi016 ceph-mon[37215]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:21.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:20 smithi016 ceph-mon[37215]: osdmap e777: 8 total, 8 up, 8 in 2024-03-26T04:50:22.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:22 smithi183 ceph-mon[33745]: osdmap e778: 8 total, 8 up, 8 in 2024-03-26T04:50:22.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:22 smithi183 ceph-mon[33745]: pgmap v1686: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 825 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:50:22.409 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:22 smithi016 ceph-mon[37215]: osdmap e778: 8 total, 8 up, 8 in 2024-03-26T04:50:22.409 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:22 smithi016 ceph-mon[37215]: pgmap v1686: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 825 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:50:22.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:22 smithi016 ceph-mon[29101]: osdmap e778: 8 total, 8 up, 8 in 2024-03-26T04:50:22.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:22 smithi016 ceph-mon[29101]: pgmap v1686: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 825 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:50:23.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:23 smithi183 ceph-mon[33745]: osdmap e779: 8 total, 8 up, 8 in 2024-03-26T04:50:23.409 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:23 smithi016 ceph-mon[37215]: osdmap e779: 8 total, 8 up, 8 in 2024-03-26T04:50:23.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:23 smithi016 ceph-mon[29101]: osdmap e779: 8 total, 8 up, 8 in 2024-03-26T04:50:24.013 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:50:23 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:50:24.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:24 smithi183 ceph-mon[33745]: osdmap e780: 8 total, 8 up, 8 in 2024-03-26T04:50:24.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:24 smithi183 ceph-mon[33745]: pgmap v1689: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 826 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:24.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:24 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:24.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:24 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:50:24.409 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:24 smithi016 ceph-mon[37215]: osdmap e780: 8 total, 8 up, 8 in 2024-03-26T04:50:24.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:24 smithi016 ceph-mon[37215]: pgmap v1689: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 826 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:24.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:24 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:24.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:24 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:50:24.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:24 smithi016 ceph-mon[29101]: osdmap e780: 8 total, 8 up, 8 in 2024-03-26T04:50:24.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:24 smithi016 ceph-mon[29101]: pgmap v1689: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 826 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:24.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:24 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:24.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:24 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:50:25.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:25 smithi183 ceph-mon[33745]: osdmap e781: 8 total, 8 up, 8 in 2024-03-26T04:50:25.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:25 smithi016 ceph-mon[29101]: osdmap e781: 8 total, 8 up, 8 in 2024-03-26T04:50:25.409 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:25 smithi016 ceph-mon[37215]: osdmap e781: 8 total, 8 up, 8 in 2024-03-26T04:50:26.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:26 smithi183 ceph-mon[33745]: osdmap e782: 8 total, 8 up, 8 in 2024-03-26T04:50:26.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:26 smithi183 ceph-mon[33745]: pgmap v1692: 164 pgs: 164 active+clean; 582 KiB data, 826 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:26.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:26 smithi016 ceph-mon[29101]: osdmap e782: 8 total, 8 up, 8 in 2024-03-26T04:50:26.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:26 smithi016 ceph-mon[29101]: pgmap v1692: 164 pgs: 164 active+clean; 582 KiB data, 826 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:26.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:26 smithi016 ceph-mon[37215]: osdmap e782: 8 total, 8 up, 8 in 2024-03-26T04:50:26.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:26 smithi016 ceph-mon[37215]: pgmap v1692: 164 pgs: 164 active+clean; 582 KiB data, 826 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:27.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:27 smithi183 ceph-mon[33745]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:27.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:27 smithi183 ceph-mon[33745]: osdmap e783: 8 total, 8 up, 8 in 2024-03-26T04:50:27.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:27 smithi016 ceph-mon[29101]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:27.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:27 smithi016 ceph-mon[29101]: osdmap e783: 8 total, 8 up, 8 in 2024-03-26T04:50:27.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:27 smithi016 ceph-mon[37215]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:27.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:27 smithi016 ceph-mon[37215]: osdmap e783: 8 total, 8 up, 8 in 2024-03-26T04:50:28.288 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:28 smithi183 ceph-mon[33745]: osdmap e784: 8 total, 8 up, 8 in 2024-03-26T04:50:28.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:28 smithi183 ceph-mon[33745]: pgmap v1695: 196 pgs: 32 creating+peering, 164 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:28.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:28 smithi016 ceph-mon[29101]: osdmap e784: 8 total, 8 up, 8 in 2024-03-26T04:50:28.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:28 smithi016 ceph-mon[29101]: pgmap v1695: 196 pgs: 32 creating+peering, 164 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:28.409 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:28 smithi016 ceph-mon[37215]: osdmap e784: 8 total, 8 up, 8 in 2024-03-26T04:50:28.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:28 smithi016 ceph-mon[37215]: pgmap v1695: 196 pgs: 32 creating+peering, 164 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:29.035 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:50:28 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:50:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:50:29.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:29 smithi183 ceph-mon[33745]: osdmap e785: 8 total, 8 up, 8 in 2024-03-26T04:50:29.409 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:29 smithi016 ceph-mon[37215]: osdmap e785: 8 total, 8 up, 8 in 2024-03-26T04:50:29.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:29 smithi016 ceph-mon[29101]: osdmap e785: 8 total, 8 up, 8 in 2024-03-26T04:50:30.310 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:30 smithi016 ceph-mon[29101]: osdmap e786: 8 total, 8 up, 8 in 2024-03-26T04:50:30.310 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:30 smithi016 ceph-mon[29101]: pgmap v1698: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:30.310 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:30 smithi016 ceph-mon[37215]: osdmap e786: 8 total, 8 up, 8 in 2024-03-26T04:50:30.310 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:30 smithi016 ceph-mon[37215]: pgmap v1698: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:30.538 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:30 smithi183 ceph-mon[33745]: osdmap e786: 8 total, 8 up, 8 in 2024-03-26T04:50:30.539 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:30 smithi183 ceph-mon[33745]: pgmap v1698: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:31.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:31 smithi016 ceph-mon[29101]: osdmap e787: 8 total, 8 up, 8 in 2024-03-26T04:50:31.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:31 smithi016 ceph-mon[37215]: osdmap e787: 8 total, 8 up, 8 in 2024-03-26T04:50:31.538 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:31 smithi183 ceph-mon[33745]: osdmap e787: 8 total, 8 up, 8 in 2024-03-26T04:50:32.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:32 smithi016 ceph-mon[29101]: osdmap e788: 8 total, 8 up, 8 in 2024-03-26T04:50:32.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:32 smithi016 ceph-mon[29101]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:32.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:32 smithi016 ceph-mon[29101]: pgmap v1701: 164 pgs: 164 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:50:32.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:32 smithi016 ceph-mon[37215]: osdmap e788: 8 total, 8 up, 8 in 2024-03-26T04:50:32.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:32 smithi016 ceph-mon[37215]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:32.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:32 smithi016 ceph-mon[37215]: pgmap v1701: 164 pgs: 164 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:50:32.538 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:32 smithi183 ceph-mon[33745]: osdmap e788: 8 total, 8 up, 8 in 2024-03-26T04:50:32.539 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:32 smithi183 ceph-mon[33745]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:32.539 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:32 smithi183 ceph-mon[33745]: pgmap v1701: 164 pgs: 164 active+clean; 582 KiB data, 844 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:50:33.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:33 smithi016 ceph-mon[29101]: osdmap e789: 8 total, 8 up, 8 in 2024-03-26T04:50:33.409 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:33 smithi016 ceph-mon[37215]: osdmap e789: 8 total, 8 up, 8 in 2024-03-26T04:50:33.440 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:33 smithi183 ceph-mon[33745]: osdmap e789: 8 total, 8 up, 8 in 2024-03-26T04:50:33.789 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:50:33 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:50:34.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:34 smithi016 ceph-mon[29101]: osdmap e790: 8 total, 8 up, 8 in 2024-03-26T04:50:34.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:34 smithi016 ceph-mon[29101]: pgmap v1704: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 852 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:34.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:34 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:34.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:34 smithi016 ceph-mon[37215]: osdmap e790: 8 total, 8 up, 8 in 2024-03-26T04:50:34.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:34 smithi016 ceph-mon[37215]: pgmap v1704: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 852 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:34.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:34 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:34.460 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:34 smithi183 ceph-mon[33745]: osdmap e790: 8 total, 8 up, 8 in 2024-03-26T04:50:34.460 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:34 smithi183 ceph-mon[33745]: pgmap v1704: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 852 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:34.460 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:34 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:35.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:35 smithi016 ceph-mon[29101]: osdmap e791: 8 total, 8 up, 8 in 2024-03-26T04:50:35.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:35 smithi016 ceph-mon[37215]: osdmap e791: 8 total, 8 up, 8 in 2024-03-26T04:50:35.538 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:35 smithi183 ceph-mon[33745]: osdmap e791: 8 total, 8 up, 8 in 2024-03-26T04:50:36.409 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:36 smithi016 ceph-mon[37215]: osdmap e792: 8 total, 8 up, 8 in 2024-03-26T04:50:36.409 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:36 smithi016 ceph-mon[37215]: pgmap v1707: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 852 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:36.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:36 smithi016 ceph-mon[29101]: osdmap e792: 8 total, 8 up, 8 in 2024-03-26T04:50:36.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:36 smithi016 ceph-mon[29101]: pgmap v1707: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 852 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:36.539 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:36 smithi183 ceph-mon[33745]: osdmap e792: 8 total, 8 up, 8 in 2024-03-26T04:50:36.539 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:36 smithi183 ceph-mon[33745]: pgmap v1707: 196 pgs: 32 unknown, 164 active+clean; 582 KiB data, 852 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:37.110 INFO:tasks.workunit.client.0.smithi016.stdout: io: Running main() from gmock_main.cc 2024-03-26T04:50:37.110 INFO:tasks.workunit.client.0.smithi016.stdout: io: [==========] Running 14 tests from 1 test suite. 2024-03-26T04:50:37.110 INFO:tasks.workunit.client.0.smithi016.stdout: io: [----------] Global test environment set-up. 2024-03-26T04:50:37.110 INFO:tasks.workunit.client.0.smithi016.stdout: io: [----------] 14 tests from NeoRadosIo 2024-03-26T04:50:37.110 INFO:tasks.workunit.client.0.smithi016.stdout: io: [ RUN ] NeoRadosIo.Limits 2024-03-26T04:50:37.110 INFO:tasks.workunit.client.0.smithi016.stdout: io: [ OK ] NeoRadosIo.Limits (1808903 ms) 2024-03-26T04:50:37.110 INFO:tasks.workunit.client.0.smithi016.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2024-03-26T04:50:37.110 INFO:tasks.workunit.client.0.smithi016.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (3028 ms) 2024-03-26T04:50:37.110 INFO:tasks.workunit.client.0.smithi016.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2024-03-26T04:50:37.110 INFO:tasks.workunit.client.0.smithi016.stdout: io: [ OK ] NeoRadosIo.ReadOp (3028 ms) 2024-03-26T04:50:37.111 INFO:tasks.workunit.client.0.smithi016.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2024-03-26T04:50:37.111 INFO:tasks.workunit.client.0.smithi016.stdout: io: [ OK ] NeoRadosIo.SparseRead (3013 ms) 2024-03-26T04:50:37.111 INFO:tasks.workunit.client.0.smithi016.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2024-03-26T04:50:37.111 INFO:tasks.workunit.client.0.smithi016.stdout: io: [ OK ] NeoRadosIo.RoundTrip (3020 ms) 2024-03-26T04:50:37.111 INFO:tasks.workunit.client.0.smithi016.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2024-03-26T04:50:37.111 INFO:tasks.workunit.client.0.smithi016.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (3019 ms) 2024-03-26T04:50:37.111 INFO:tasks.workunit.client.0.smithi016.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2024-03-26T04:50:37.111 INFO:tasks.workunit.client.0.smithi016.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (3018 ms) 2024-03-26T04:50:37.111 INFO:tasks.workunit.client.0.smithi016.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2024-03-26T04:50:37.111 INFO:tasks.workunit.client.0.smithi016.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (3020 ms) 2024-03-26T04:50:37.111 INFO:tasks.workunit.client.0.smithi016.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2024-03-26T04:50:37.111 INFO:tasks.workunit.client.0.smithi016.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (3025 ms) 2024-03-26T04:50:37.111 INFO:tasks.workunit.client.0.smithi016.stdout: io: [ RUN ] NeoRadosIo.Trunc 2024-03-26T04:50:37.111 INFO:tasks.workunit.client.0.smithi016.stdout: io: [ OK ] NeoRadosIo.Trunc (3019 ms) 2024-03-26T04:50:37.112 INFO:tasks.workunit.client.0.smithi016.stdout: io: [ RUN ] NeoRadosIo.Remove 2024-03-26T04:50:37.112 INFO:tasks.workunit.client.0.smithi016.stdout: io: [ OK ] NeoRadosIo.Remove (3024 ms) 2024-03-26T04:50:37.112 INFO:tasks.workunit.client.0.smithi016.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2024-03-26T04:50:37.112 INFO:tasks.workunit.client.0.smithi016.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (3011 ms) 2024-03-26T04:50:37.112 INFO:tasks.workunit.client.0.smithi016.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2024-03-26T04:50:37.112 INFO:tasks.workunit.client.0.smithi016.stdout: io: [ OK ] NeoRadosIo.RmXattr (3026 ms) 2024-03-26T04:50:37.112 INFO:tasks.workunit.client.0.smithi016.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2024-03-26T04:50:37.112 INFO:tasks.workunit.client.0.smithi016.stdout: io: [ OK ] NeoRadosIo.GetXattrs (3023 ms) 2024-03-26T04:50:37.112 INFO:tasks.workunit.client.0.smithi016.stdout: io: [----------] 14 tests from NeoRadosIo (1848177 ms total) 2024-03-26T04:50:37.112 INFO:tasks.workunit.client.0.smithi016.stdout: io: 2024-03-26T04:50:37.112 INFO:tasks.workunit.client.0.smithi016.stdout: io: [----------] Global test environment tear-down 2024-03-26T04:50:37.112 INFO:tasks.workunit.client.0.smithi016.stdout: io: [==========] 14 tests from 1 test suite ran. (1848177 ms total) 2024-03-26T04:50:37.112 INFO:tasks.workunit.client.0.smithi016.stdout: io: [ PASSED ] 14 tests. 2024-03-26T04:50:37.118 INFO:tasks.workunit.client.0.smithi016.stderr:+ for t in "${!pids[@]}" 2024-03-26T04:50:37.118 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=75290 2024-03-26T04:50:37.118 INFO:tasks.workunit.client.0.smithi016.stderr:+ wait 75290 2024-03-26T04:50:37.118 INFO:tasks.workunit.client.0.smithi016.stderr:+ for t in "${!pids[@]}" 2024-03-26T04:50:37.118 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74343 2024-03-26T04:50:37.118 INFO:tasks.workunit.client.0.smithi016.stderr:+ wait 74343 2024-03-26T04:50:37.118 INFO:tasks.workunit.client.0.smithi016.stderr:+ for t in "${!pids[@]}" 2024-03-26T04:50:37.119 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=75055 2024-03-26T04:50:37.119 INFO:tasks.workunit.client.0.smithi016.stderr:+ wait 75055 2024-03-26T04:50:37.119 INFO:tasks.workunit.client.0.smithi016.stderr:+ for t in "${!pids[@]}" 2024-03-26T04:50:37.119 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74896 2024-03-26T04:50:37.119 INFO:tasks.workunit.client.0.smithi016.stderr:+ wait 74896 2024-03-26T04:50:37.119 INFO:tasks.workunit.client.0.smithi016.stderr:+ for t in "${!pids[@]}" 2024-03-26T04:50:37.119 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74209 2024-03-26T04:50:37.119 INFO:tasks.workunit.client.0.smithi016.stderr:+ wait 74209 2024-03-26T04:50:37.119 INFO:tasks.workunit.client.0.smithi016.stderr:+ for t in "${!pids[@]}" 2024-03-26T04:50:37.120 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74270 2024-03-26T04:50:37.120 INFO:tasks.workunit.client.0.smithi016.stderr:+ wait 74270 2024-03-26T04:50:37.120 INFO:tasks.workunit.client.0.smithi016.stderr:+ for t in "${!pids[@]}" 2024-03-26T04:50:37.120 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74920 2024-03-26T04:50:37.120 INFO:tasks.workunit.client.0.smithi016.stderr:+ wait 74920 2024-03-26T04:50:37.120 INFO:tasks.workunit.client.0.smithi016.stderr:+ for t in "${!pids[@]}" 2024-03-26T04:50:37.120 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74200 2024-03-26T04:50:37.120 INFO:tasks.workunit.client.0.smithi016.stderr:+ wait 74200 2024-03-26T04:50:37.120 INFO:tasks.workunit.client.0.smithi016.stderr:+ for t in "${!pids[@]}" 2024-03-26T04:50:37.120 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74220 2024-03-26T04:50:37.120 INFO:tasks.workunit.client.0.smithi016.stderr:+ wait 74220 2024-03-26T04:50:37.120 INFO:tasks.workunit.client.0.smithi016.stderr:+ for t in "${!pids[@]}" 2024-03-26T04:50:37.120 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=75363 2024-03-26T04:50:37.120 INFO:tasks.workunit.client.0.smithi016.stderr:+ wait 75363 2024-03-26T04:50:37.120 INFO:tasks.workunit.client.0.smithi016.stderr:+ for t in "${!pids[@]}" 2024-03-26T04:50:37.121 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74484 2024-03-26T04:50:37.121 INFO:tasks.workunit.client.0.smithi016.stderr:+ wait 74484 2024-03-26T04:50:37.121 INFO:tasks.workunit.client.0.smithi016.stderr:+ for t in "${!pids[@]}" 2024-03-26T04:50:37.121 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=75484 2024-03-26T04:50:37.121 INFO:tasks.workunit.client.0.smithi016.stderr:+ wait 75484 2024-03-26T04:50:37.121 INFO:tasks.workunit.client.0.smithi016.stderr:+ for t in "${!pids[@]}" 2024-03-26T04:50:37.121 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74624 2024-03-26T04:50:37.121 INFO:tasks.workunit.client.0.smithi016.stderr:+ wait 74624 2024-03-26T04:50:37.121 INFO:tasks.workunit.client.0.smithi016.stderr:+ for t in "${!pids[@]}" 2024-03-26T04:50:37.121 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=75209 2024-03-26T04:50:37.121 INFO:tasks.workunit.client.0.smithi016.stderr:+ wait 75209 2024-03-26T04:50:37.121 INFO:tasks.workunit.client.0.smithi016.stderr:+ for t in "${!pids[@]}" 2024-03-26T04:50:37.121 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=75535 2024-03-26T04:50:37.121 INFO:tasks.workunit.client.0.smithi016.stderr:+ wait 75535 2024-03-26T04:50:37.122 INFO:tasks.workunit.client.0.smithi016.stderr:+ for t in "${!pids[@]}" 2024-03-26T04:50:37.122 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74538 2024-03-26T04:50:37.122 INFO:tasks.workunit.client.0.smithi016.stderr:+ wait 74538 2024-03-26T04:50:37.122 INFO:tasks.workunit.client.0.smithi016.stderr:+ for t in "${!pids[@]}" 2024-03-26T04:50:37.122 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74980 2024-03-26T04:50:37.122 INFO:tasks.workunit.client.0.smithi016.stderr:+ wait 74980 2024-03-26T04:50:37.122 INFO:tasks.workunit.client.0.smithi016.stderr:+ for t in "${!pids[@]}" 2024-03-26T04:50:37.122 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=75328 2024-03-26T04:50:37.122 INFO:tasks.workunit.client.0.smithi016.stderr:+ wait 75328 2024-03-26T04:50:37.122 INFO:tasks.workunit.client.0.smithi016.stderr:+ for t in "${!pids[@]}" 2024-03-26T04:50:37.122 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=75075 2024-03-26T04:50:37.122 INFO:tasks.workunit.client.0.smithi016.stderr:+ wait 75075 2024-03-26T04:50:37.122 INFO:tasks.workunit.client.0.smithi016.stderr:+ for t in "${!pids[@]}" 2024-03-26T04:50:37.122 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=75441 2024-03-26T04:50:37.123 INFO:tasks.workunit.client.0.smithi016.stderr:+ wait 75441 2024-03-26T04:50:37.123 INFO:tasks.workunit.client.0.smithi016.stderr:+ for t in "${!pids[@]}" 2024-03-26T04:50:37.123 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74863 2024-03-26T04:50:37.123 INFO:tasks.workunit.client.0.smithi016.stderr:+ wait 74863 2024-03-26T04:50:37.123 INFO:tasks.workunit.client.0.smithi016.stderr:+ for t in "${!pids[@]}" 2024-03-26T04:50:37.123 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74399 2024-03-26T04:50:37.123 INFO:tasks.workunit.client.0.smithi016.stderr:+ wait 74399 2024-03-26T04:50:37.123 INFO:tasks.workunit.client.0.smithi016.stderr:+ for t in "${!pids[@]}" 2024-03-26T04:50:37.123 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=75521 2024-03-26T04:50:37.123 INFO:tasks.workunit.client.0.smithi016.stderr:+ wait 75521 2024-03-26T04:50:37.123 INFO:tasks.workunit.client.0.smithi016.stderr:+ for t in "${!pids[@]}" 2024-03-26T04:50:37.123 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74743 2024-03-26T04:50:37.123 INFO:tasks.workunit.client.0.smithi016.stderr:+ wait 74743 2024-03-26T04:50:37.123 INFO:tasks.workunit.client.0.smithi016.stderr:+ for t in "${!pids[@]}" 2024-03-26T04:50:37.124 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74193 2024-03-26T04:50:37.124 INFO:tasks.workunit.client.0.smithi016.stderr:+ wait 74193 2024-03-26T04:50:37.124 INFO:tasks.workunit.client.0.smithi016.stderr:+ for t in "${!pids[@]}" 2024-03-26T04:50:37.124 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74250 2024-03-26T04:50:37.124 INFO:tasks.workunit.client.0.smithi016.stderr:+ wait 74250 2024-03-26T04:50:37.124 INFO:tasks.workunit.client.0.smithi016.stderr:+ for t in "${!pids[@]}" 2024-03-26T04:50:37.124 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74638 2024-03-26T04:50:37.124 INFO:tasks.workunit.client.0.smithi016.stderr:+ wait 74638 2024-03-26T04:50:37.124 INFO:tasks.workunit.client.0.smithi016.stderr:+ for t in "${!pids[@]}" 2024-03-26T04:50:37.124 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74301 2024-03-26T04:50:37.124 INFO:tasks.workunit.client.0.smithi016.stderr:+ wait 74301 2024-03-26T04:50:37.124 INFO:tasks.workunit.client.0.smithi016.stderr:+ for t in "${!pids[@]}" 2024-03-26T04:50:37.124 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74770 2024-03-26T04:50:37.124 INFO:tasks.workunit.client.0.smithi016.stderr:+ wait 74770 2024-03-26T04:50:37.124 INFO:tasks.workunit.client.0.smithi016.stderr:+ for t in "${!pids[@]}" 2024-03-26T04:50:37.125 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=74823 2024-03-26T04:50:37.125 INFO:tasks.workunit.client.0.smithi016.stderr:+ wait 74823 2024-03-26T04:50:37.125 INFO:tasks.workunit.client.0.smithi016.stderr:+ for t in "${!pids[@]}" 2024-03-26T04:50:37.125 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=75303 2024-03-26T04:50:37.125 INFO:tasks.workunit.client.0.smithi016.stderr:+ wait 75303 2024-03-26T04:50:37.125 INFO:tasks.workunit.client.0.smithi016.stderr:+ for t in "${!pids[@]}" 2024-03-26T04:50:37.125 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=75583 2024-03-26T04:50:37.125 INFO:tasks.workunit.client.0.smithi016.stderr:+ wait 75583 2024-03-26T04:50:37.125 INFO:tasks.workunit.client.0.smithi016.stderr:+ exit 0 2024-03-26T04:50:37.125 INFO:tasks.workunit.client.0.smithi016.stderr:+ cleanup 2024-03-26T04:50:37.125 INFO:tasks.workunit.client.0.smithi016.stderr:+ pkill -P 74187 2024-03-26T04:50:37.133 INFO:tasks.workunit.client.0.smithi016.stderr:+ true 2024-03-26T04:50:37.133 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-26T04:50:37.133 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-26T04:50:37.168 INFO:tasks.workunit:Running workunits matching rados/test_pool_quota.sh on client.0... 2024-03-26T04:50:37.169 INFO:tasks.workunit:Running workunit rados/test_pool_quota.sh... 2024-03-26T04:50:37.169 DEBUG:teuthology.orchestra.run.smithi016:workunit test rados/test_pool_quota.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=9f04c91b77e4619963d41730d8cf756e3a6152de TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test_pool_quota.sh 2024-03-26T04:50:37.229 INFO:tasks.workunit.client.0.smithi016.stderr:++ uuidgen 2024-03-26T04:50:37.237 INFO:tasks.workunit.client.0.smithi016.stderr:+ p=9afc82fb-6d6b-4f10-901a-c3cc23ea025f 2024-03-26T04:50:37.238 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph osd pool create 9afc82fb-6d6b-4f10-901a-c3cc23ea025f 12 2024-03-26T04:50:37.327 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:37.328+0000 7f7c8289d640 1 -- 172.21.15.16:0/2874046800 >> v1:172.21.15.16:6790/0 conn(0x7f7c7c10a290 legacy=0x7f7c7c10c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:50:37.327 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:37.328+0000 7f7c8289d640 1 -- 172.21.15.16:0/2874046800 shutdown_connections 2024-03-26T04:50:37.327 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:37.328+0000 7f7c8289d640 1 -- 172.21.15.16:0/2874046800 >> 172.21.15.16:0/2874046800 conn(0x7f7c7c0fced0 msgr2=0x7f7c7c0ff2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:50:37.327 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:37.329+0000 7f7c8289d640 1 -- 172.21.15.16:0/2874046800 shutdown_connections 2024-03-26T04:50:37.327 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:37.329+0000 7f7c8289d640 1 -- 172.21.15.16:0/2874046800 wait complete. 2024-03-26T04:50:37.327 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:37.329+0000 7f7c8289d640 1 Processor -- start 2024-03-26T04:50:37.327 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:37.329+0000 7f7c8289d640 1 -- start start 2024-03-26T04:50:37.327 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:37.329+0000 7f7c8289d640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7c7c1a6280 con 0x7f7c7c10de60 2024-03-26T04:50:37.327 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:37.329+0000 7f7c8289d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7c7c1a7480 con 0x7f7c7c10a290 2024-03-26T04:50:37.327 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:37.329+0000 7f7c8289d640 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7c7c1a8680 con 0x7f7c7c107480 2024-03-26T04:50:37.327 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:37.330+0000 7f7c7bfff640 1 --1- >> v1:172.21.15.16:6790/0 conn(0x7f7c7c107480 0x7f7c7c10d5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6790/0 says I am v1:172.21.15.16:54468/0 (socket says 172.21.15.16:54468) 2024-03-26T04:50:37.328 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:37.330+0000 7f7c7bfff640 1 -- 172.21.15.16:0/266151765 learned_addr learned my addr 172.21.15.16:0/266151765 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:50:37.330 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:37.330+0000 7f7c797fa640 1 -- 172.21.15.16:0/266151765 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1994571027 0 0) 0x7f7c7c1a8680 con 0x7f7c7c107480 2024-03-26T04:50:37.330 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:37.330+0000 7f7c797fa640 1 -- 172.21.15.16:0/266151765 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c64003520 con 0x7f7c7c107480 2024-03-26T04:50:37.330 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:37.330+0000 7f7c797fa640 1 -- 172.21.15.16:0/266151765 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 797627648 0 0) 0x7f7c7c1a6280 con 0x7f7c7c10de60 2024-03-26T04:50:37.330 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:37.330+0000 7f7c797fa640 1 -- 172.21.15.16:0/266151765 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c7c1a8680 con 0x7f7c7c10de60 2024-03-26T04:50:37.330 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:37.330+0000 7f7c797fa640 1 -- 172.21.15.16:0/266151765 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2569304712 0 0) 0x7f7c7c1a7480 con 0x7f7c7c10a290 2024-03-26T04:50:37.330 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:37.330+0000 7f7c797fa640 1 -- 172.21.15.16:0/266151765 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c7c1a6280 con 0x7f7c7c10a290 2024-03-26T04:50:37.330 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:37.330+0000 7f7c797fa640 1 -- 172.21.15.16:0/266151765 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2417395807 0 0) 0x7f7c64003520 con 0x7f7c7c107480 2024-03-26T04:50:37.331 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:37.330+0000 7f7c797fa640 1 -- 172.21.15.16:0/266151765 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c7c1a7480 con 0x7f7c7c107480 2024-03-26T04:50:37.331 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:37.330+0000 7f7c797fa640 1 -- 172.21.15.16:0/266151765 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f7c6c0018b0 con 0x7f7c7c107480 2024-03-26T04:50:37.331 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:37.330+0000 7f7c797fa640 1 -- 172.21.15.16:0/266151765 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 251824890 0 0) 0x7f7c7c1a8680 con 0x7f7c7c10de60 2024-03-26T04:50:37.331 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:37.330+0000 7f7c797fa640 1 -- 172.21.15.16:0/266151765 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c64003520 con 0x7f7c7c10de60 2024-03-26T04:50:37.331 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:37.330+0000 7f7c797fa640 1 -- 172.21.15.16:0/266151765 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f7c74001b30 con 0x7f7c7c10de60 2024-03-26T04:50:37.331 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:37.330+0000 7f7c797fa640 1 -- 172.21.15.16:0/266151765 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2399178073 0 0) 0x7f7c7c1a6280 con 0x7f7c7c10a290 2024-03-26T04:50:37.331 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:37.330+0000 7f7c797fa640 1 -- 172.21.15.16:0/266151765 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c7c1a8680 con 0x7f7c7c10a290 2024-03-26T04:50:37.331 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:37.330+0000 7f7c797fa640 1 -- 172.21.15.16:0/266151765 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f7c70004570 con 0x7f7c7c10a290 2024-03-26T04:50:37.331 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:37.330+0000 7f7c797fa640 1 -- 172.21.15.16:0/266151765 <== mon.2 v1:172.21.15.16:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1572534361 0 0) 0x7f7c7c1a7480 con 0x7f7c7c107480 2024-03-26T04:50:37.331 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:37.330+0000 7f7c797fa640 1 -- 172.21.15.16:0/266151765 >> v1:172.21.15.183:6789/0 conn(0x7f7c7c10a290 legacy=0x7f7c7c1a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:50:37.331 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:37.330+0000 7f7c797fa640 1 -- 172.21.15.16:0/266151765 >> v1:172.21.15.16:6789/0 conn(0x7f7c7c10de60 legacy=0x7f7c7c1a4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:50:37.331 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:37.330+0000 7f7c797fa640 1 -- 172.21.15.16:0/266151765 --> v1:172.21.15.16:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c7c1a9880 con 0x7f7c7c107480 2024-03-26T04:50:37.331 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:37.330+0000 7f7c8289d640 1 -- 172.21.15.16:0/266151765 --> v1:172.21.15.16:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c7c1a6470 con 0x7f7c7c107480 2024-03-26T04:50:37.331 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:37.330+0000 7f7c8289d640 1 -- 172.21.15.16:0/266151765 --> v1:172.21.15.16:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c7c1a6900 con 0x7f7c7c107480 2024-03-26T04:50:37.332 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:37.330+0000 7f7c797fa640 1 -- 172.21.15.16:0/266151765 <== mon.2 v1:172.21.15.16:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f7c6c003aa0 con 0x7f7c7c107480 2024-03-26T04:50:37.332 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:37.330+0000 7f7c797fa640 1 -- 172.21.15.16:0/266151765 <== mon.2 v1:172.21.15.16:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f7c6c005b20 con 0x7f7c7c107480 2024-03-26T04:50:37.332 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:37.331+0000 7f7c8289d640 1 -- 172.21.15.16:0/266151765 --> v1:172.21.15.16:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c680050b0 con 0x7f7c7c107480 2024-03-26T04:50:37.332 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:37.332+0000 7f7c797fa640 1 -- 172.21.15.16:0/266151765 <== mon.2 v1:172.21.15.16:6790/0 7 ==== mgrmap(e 22) v1 ==== 91420+0+0 (unknown 3095259778 0 0) 0x7f7c6c003610 con 0x7f7c7c107480 2024-03-26T04:50:37.332 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:37.332+0000 7f7c797fa640 1 -- 172.21.15.16:0/266151765 <== mon.2 v1:172.21.15.16:6790/0 8 ==== osd_map(794..794 src has 253..794) v4 ==== 7410+0+0 (unknown 2106760522 0 0) 0x7f7c6c088d50 con 0x7f7c7c107480 2024-03-26T04:50:37.335 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:37.337+0000 7f7c797fa640 1 -- 172.21.15.16:0/266151765 <== mon.2 v1:172.21.15.16:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f7c6c055b70 con 0x7f7c7c107480 2024-03-26T04:50:37.409 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:37 smithi016 ceph-mon[37215]: osdmap e793: 8 total, 8 up, 8 in 2024-03-26T04:50:37.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:37 smithi016 ceph-mon[29101]: osdmap e793: 8 total, 8 up, 8 in 2024-03-26T04:50:37.519 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:37.521+0000 7f7c8289d640 1 -- 172.21.15.16:0/266151765 --> v1:172.21.15.16:6790/0 -- mon_command({"prefix": "osd pool create", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "pg_num": 12} v 0) v1 -- 0x7f7c680058e0 con 0x7f7c7c107480 2024-03-26T04:50:37.538 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:37 smithi183 ceph-mon[33745]: osdmap e793: 8 total, 8 up, 8 in 2024-03-26T04:50:38.109 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.112+0000 7f7c797fa640 1 -- 172.21.15.16:0/266151765 <== mon.2 v1:172.21.15.16:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "pg_num": 12}]=0 pool '9afc82fb-6d6b-4f10-901a-c3cc23ea025f' created v795) v1 ==== 176+0+0 (unknown 398485413 0 0) 0x7f7c6c053ee0 con 0x7f7c7c107480 2024-03-26T04:50:38.226 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.229+0000 7f7c8289d640 1 -- 172.21.15.16:0/266151765 --> v1:172.21.15.16:6790/0 -- mon_command({"prefix": "osd pool create", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "pg_num": 12} v 0) v1 -- 0x7f7c68005aa0 con 0x7f7c7c107480 2024-03-26T04:50:38.227 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.229+0000 7f7c797fa640 1 -- 172.21.15.16:0/266151765 <== mon.2 v1:172.21.15.16:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "pg_num": 12}]=0 pool '9afc82fb-6d6b-4f10-901a-c3cc23ea025f' already exists v795) v1 ==== 183+0+0 (unknown 3456190690 0 0) 0x7f7c6c055700 con 0x7f7c7c107480 2024-03-26T04:50:38.227 INFO:tasks.workunit.client.0.smithi016.stderr:pool '9afc82fb-6d6b-4f10-901a-c3cc23ea025f' already exists 2024-03-26T04:50:38.229 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.231+0000 7f7c8289d640 1 -- 172.21.15.16:0/266151765 >> v1:172.21.15.16:6800/3868004154 conn(0x7f7c6406e9f0 legacy=0x7f7c64070eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:50:38.229 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.232+0000 7f7c8289d640 1 -- 172.21.15.16:0/266151765 >> v1:172.21.15.16:6790/0 conn(0x7f7c7c107480 legacy=0x7f7c7c10d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:50:38.229 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.232+0000 7f7c8289d640 1 -- 172.21.15.16:0/266151765 shutdown_connections 2024-03-26T04:50:38.230 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.232+0000 7f7c8289d640 1 -- 172.21.15.16:0/266151765 >> 172.21.15.16:0/266151765 conn(0x7f7c7c0fced0 msgr2=0x7f7c7c100330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:50:38.230 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.232+0000 7f7c8289d640 1 -- 172.21.15.16:0/266151765 shutdown_connections 2024-03-26T04:50:38.230 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.232+0000 7f7c8289d640 1 -- 172.21.15.16:0/266151765 wait complete. 2024-03-26T04:50:38.238 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph osd pool set-quota 9afc82fb-6d6b-4f10-901a-c3cc23ea025f max_objects 10 2024-03-26T04:50:38.321 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.323+0000 7f699a908640 1 -- 172.21.15.16:0/2855107735 >> v1:172.21.15.16:6789/0 conn(0x7f699410a2b0 legacy=0x7f699410c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:50:38.321 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.323+0000 7f699a908640 1 -- 172.21.15.16:0/2855107735 shutdown_connections 2024-03-26T04:50:38.321 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.323+0000 7f699a908640 1 -- 172.21.15.16:0/2855107735 >> 172.21.15.16:0/2855107735 conn(0x7f69940fce70 msgr2=0x7f69940ff290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:50:38.321 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.323+0000 7f699a908640 1 -- 172.21.15.16:0/2855107735 shutdown_connections 2024-03-26T04:50:38.321 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.323+0000 7f699a908640 1 -- 172.21.15.16:0/2855107735 wait complete. 2024-03-26T04:50:38.321 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.324+0000 7f699a908640 1 Processor -- start 2024-03-26T04:50:38.321 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.324+0000 7f699a908640 1 -- start start 2024-03-26T04:50:38.324 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.324+0000 7f699a908640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f69941a6260 con 0x7f699410a2b0 2024-03-26T04:50:38.324 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.324+0000 7f699a908640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f69941a7460 con 0x7f6994101630 2024-03-26T04:50:38.324 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.324+0000 7f699a908640 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f69941a8660 con 0x7f699410de20 2024-03-26T04:50:38.324 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.324+0000 7f6998ea5640 1 --1- >> v1:172.21.15.16:6790/0 conn(0x7f699410de20 0x7f69941a4960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6790/0 says I am v1:172.21.15.16:54482/0 (socket says 172.21.15.16:54482) 2024-03-26T04:50:38.324 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.324+0000 7f69937fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f699410a2b0 0x7f69941a1190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:54988/0 (socket says 172.21.15.16:54988) 2024-03-26T04:50:38.324 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.324+0000 7f6998ea5640 1 -- 172.21.15.16:0/918547564 learned_addr learned my addr 172.21.15.16:0/918547564 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:50:38.324 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.324+0000 7f69917fa640 1 -- 172.21.15.16:0/918547564 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2808153575 0 0) 0x7f69941a8660 con 0x7f699410de20 2024-03-26T04:50:38.324 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.324+0000 7f69917fa640 1 -- 172.21.15.16:0/918547564 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f697c003520 con 0x7f699410de20 2024-03-26T04:50:38.325 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.324+0000 7f69917fa640 1 -- 172.21.15.16:0/918547564 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3046346610 0 0) 0x7f69941a6260 con 0x7f699410a2b0 2024-03-26T04:50:38.325 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.324+0000 7f69917fa640 1 -- 172.21.15.16:0/918547564 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69941a8660 con 0x7f699410a2b0 2024-03-26T04:50:38.325 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.325+0000 7f69917fa640 1 -- 172.21.15.16:0/918547564 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1387867277 0 0) 0x7f697c003520 con 0x7f699410de20 2024-03-26T04:50:38.325 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.325+0000 7f69917fa640 1 -- 172.21.15.16:0/918547564 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69941a6260 con 0x7f699410de20 2024-03-26T04:50:38.325 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.325+0000 7f69917fa640 1 -- 172.21.15.16:0/918547564 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f698c001c70 con 0x7f699410de20 2024-03-26T04:50:38.325 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.325+0000 7f69917fa640 1 -- 172.21.15.16:0/918547564 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 170205760 0 0) 0x7f69941a8660 con 0x7f699410a2b0 2024-03-26T04:50:38.325 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.325+0000 7f69917fa640 1 -- 172.21.15.16:0/918547564 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f697c003520 con 0x7f699410a2b0 2024-03-26T04:50:38.325 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.325+0000 7f69917fa640 1 -- 172.21.15.16:0/918547564 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f6988004330 con 0x7f699410a2b0 2024-03-26T04:50:38.325 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.325+0000 7f69917fa640 1 -- 172.21.15.16:0/918547564 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3169038692 0 0) 0x7f69941a7460 con 0x7f6994101630 2024-03-26T04:50:38.325 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.325+0000 7f69917fa640 1 -- 172.21.15.16:0/918547564 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69941a8660 con 0x7f6994101630 2024-03-26T04:50:38.325 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.325+0000 7f69917fa640 1 -- 172.21.15.16:0/918547564 <== mon.2 v1:172.21.15.16:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3718612602 0 0) 0x7f69941a6260 con 0x7f699410de20 2024-03-26T04:50:38.325 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.325+0000 7f69917fa640 1 -- 172.21.15.16:0/918547564 >> v1:172.21.15.183:6789/0 conn(0x7f6994101630 legacy=0x7f6994106880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:50:38.325 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.325+0000 7f69917fa640 1 -- 172.21.15.16:0/918547564 >> v1:172.21.15.16:6789/0 conn(0x7f699410a2b0 legacy=0x7f69941a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:50:38.325 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.325+0000 7f69917fa640 1 -- 172.21.15.16:0/918547564 --> v1:172.21.15.16:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69941a9860 con 0x7f699410de20 2024-03-26T04:50:38.325 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.325+0000 7f699a908640 1 -- 172.21.15.16:0/918547564 --> v1:172.21.15.16:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69941a6450 con 0x7f699410de20 2024-03-26T04:50:38.326 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.325+0000 7f699a908640 1 -- 172.21.15.16:0/918547564 --> v1:172.21.15.16:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f69941a68e0 con 0x7f699410de20 2024-03-26T04:50:38.326 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.325+0000 7f69917fa640 1 -- 172.21.15.16:0/918547564 <== mon.2 v1:172.21.15.16:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f698c002000 con 0x7f699410de20 2024-03-26T04:50:38.326 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.325+0000 7f69917fa640 1 -- 172.21.15.16:0/918547564 <== mon.2 v1:172.21.15.16:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f698c006020 con 0x7f699410de20 2024-03-26T04:50:38.326 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.326+0000 7f699a908640 1 -- 172.21.15.16:0/918547564 --> v1:172.21.15.16:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6980005080 con 0x7f699410de20 2024-03-26T04:50:38.326 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.326+0000 7f69917fa640 1 -- 172.21.15.16:0/918547564 <== mon.2 v1:172.21.15.16:6790/0 7 ==== mgrmap(e 22) v1 ==== 91420+0+0 (unknown 3095259778 0 0) 0x7f698c01c6d0 con 0x7f699410de20 2024-03-26T04:50:38.326 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.327+0000 7f69917fa640 1 -- 172.21.15.16:0/918547564 <== mon.2 v1:172.21.15.16:6790/0 8 ==== osd_map(795..795 src has 253..795) v4 ==== 7785+0+0 (unknown 1058245127 0 0) 0x7f698c089300 con 0x7f699410de20 2024-03-26T04:50:38.328 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.331+0000 7f69917fa640 1 -- 172.21.15.16:0/918547564 <== mon.2 v1:172.21.15.16:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f698c055ff0 con 0x7f699410de20 2024-03-26T04:50:38.409 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:38 smithi016 ceph-mon[37215]: osdmap e794: 8 total, 8 up, 8 in 2024-03-26T04:50:38.409 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:38 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/266151765' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "pg_num": 12}]: dispatch 2024-03-26T04:50:38.409 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:38 smithi016 ceph-mon[37215]: from='client.56807 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "pg_num": 12}]: dispatch 2024-03-26T04:50:38.409 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:38 smithi016 ceph-mon[37215]: pgmap v1710: 164 pgs: 164 active+clean; 582 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:38.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:38 smithi016 ceph-mon[29101]: osdmap e794: 8 total, 8 up, 8 in 2024-03-26T04:50:38.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:38 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/266151765' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "pg_num": 12}]: dispatch 2024-03-26T04:50:38.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:38 smithi016 ceph-mon[29101]: from='client.56807 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "pg_num": 12}]: dispatch 2024-03-26T04:50:38.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:38 smithi016 ceph-mon[29101]: pgmap v1710: 164 pgs: 164 active+clean; 582 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:38.512 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:38.515+0000 7f699a908640 1 -- 172.21.15.16:0/918547564 --> v1:172.21.15.16:6790/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "10"} v 0) v1 -- 0x7f6980005330 con 0x7f699410de20 2024-03-26T04:50:38.538 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:38 smithi183 ceph-mon[33745]: osdmap e794: 8 total, 8 up, 8 in 2024-03-26T04:50:38.538 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:38 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/266151765' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "pg_num": 12}]: dispatch 2024-03-26T04:50:38.539 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:38 smithi183 ceph-mon[33745]: from='client.56807 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "pg_num": 12}]: dispatch 2024-03-26T04:50:38.539 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:38 smithi183 ceph-mon[33745]: pgmap v1710: 164 pgs: 164 active+clean; 582 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:39.107 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:50:38 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:50:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:50:39.114 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:39.117+0000 7f69917fa640 1 -- 172.21.15.16:0/918547564 <== mon.2 v1:172.21.15.16:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool 9afc82fb-6d6b-4f10-901a-c3cc23ea025f v796) v1 ==== 223+0+0 (unknown 3697980171 0 0) 0x7f698c054360 con 0x7f699410de20 2024-03-26T04:50:39.232 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:39.235+0000 7f699a908640 1 -- 172.21.15.16:0/918547564 --> v1:172.21.15.16:6790/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "10"} v 0) v1 -- 0x7f6980001090 con 0x7f699410de20 2024-03-26T04:50:39.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:39 smithi016 ceph-mon[29101]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:39.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:39 smithi016 ceph-mon[29101]: from='client.56807 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "pg_num": 12}]': finished 2024-03-26T04:50:39.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:39 smithi016 ceph-mon[29101]: osdmap e795: 8 total, 8 up, 8 in 2024-03-26T04:50:39.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:39 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/266151765' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "pg_num": 12}]: dispatch 2024-03-26T04:50:39.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:39 smithi016 ceph-mon[29101]: from='client.56807 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "pg_num": 12}]: dispatch 2024-03-26T04:50:39.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:39 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/918547564' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T04:50:39.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:39 smithi016 ceph-mon[29101]: from='client.56813 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T04:50:39.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:39 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:50:39.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:39 smithi016 ceph-mon[37215]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:39.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:39 smithi016 ceph-mon[37215]: from='client.56807 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "pg_num": 12}]': finished 2024-03-26T04:50:39.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:39 smithi016 ceph-mon[37215]: osdmap e795: 8 total, 8 up, 8 in 2024-03-26T04:50:39.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:39 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/266151765' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "pg_num": 12}]: dispatch 2024-03-26T04:50:39.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:39 smithi016 ceph-mon[37215]: from='client.56807 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "pg_num": 12}]: dispatch 2024-03-26T04:50:39.411 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:39 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/918547564' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T04:50:39.411 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:39 smithi016 ceph-mon[37215]: from='client.56813 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T04:50:39.411 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:39 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:50:39.538 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:39 smithi183 ceph-mon[33745]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:50:39.539 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:39 smithi183 ceph-mon[33745]: from='client.56807 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "pg_num": 12}]': finished 2024-03-26T04:50:39.539 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:39 smithi183 ceph-mon[33745]: osdmap e795: 8 total, 8 up, 8 in 2024-03-26T04:50:39.539 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:39 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/266151765' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "pg_num": 12}]: dispatch 2024-03-26T04:50:39.539 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:39 smithi183 ceph-mon[33745]: from='client.56807 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "pg_num": 12}]: dispatch 2024-03-26T04:50:39.539 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:39 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/918547564' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T04:50:39.539 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:39 smithi183 ceph-mon[33745]: from='client.56813 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T04:50:39.539 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:39 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:50:40.121 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.124+0000 7f69917fa640 1 -- 172.21.15.16:0/918547564 <== mon.2 v1:172.21.15.16:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool 9afc82fb-6d6b-4f10-901a-c3cc23ea025f v797) v1 ==== 223+0+0 (unknown 2502891131 0 0) 0x7f698c055b80 con 0x7f699410de20 2024-03-26T04:50:40.121 INFO:tasks.workunit.client.0.smithi016.stderr:set-quota max_objects = 10 for pool 9afc82fb-6d6b-4f10-901a-c3cc23ea025f 2024-03-26T04:50:40.124 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.127+0000 7f699a908640 1 -- 172.21.15.16:0/918547564 >> v1:172.21.15.16:6800/3868004154 conn(0x7f697c06e960 legacy=0x7f697c070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:50:40.124 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.127+0000 7f699a908640 1 -- 172.21.15.16:0/918547564 >> v1:172.21.15.16:6790/0 conn(0x7f699410de20 legacy=0x7f69941a4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:50:40.124 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.127+0000 7f699a908640 1 -- 172.21.15.16:0/918547564 shutdown_connections 2024-03-26T04:50:40.125 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.127+0000 7f699a908640 1 -- 172.21.15.16:0/918547564 >> 172.21.15.16:0/918547564 conn(0x7f69940fce70 msgr2=0x7f6994111320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:50:40.125 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.127+0000 7f699a908640 1 -- 172.21.15.16:0/918547564 shutdown_connections 2024-03-26T04:50:40.125 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.127+0000 7f699a908640 1 -- 172.21.15.16:0/918547564 wait complete. 2024-03-26T04:50:40.135 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph osd pool application enable 9afc82fb-6d6b-4f10-901a-c3cc23ea025f rados 2024-03-26T04:50:40.224 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.227+0000 7fbca7eed640 1 -- 172.21.15.16:0/1474525423 >> v1:172.21.15.16:6790/0 conn(0x7fbca010df00 legacy=0x7fbca01102f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:50:40.225 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.227+0000 7fbca7eed640 1 -- 172.21.15.16:0/1474525423 shutdown_connections 2024-03-26T04:50:40.225 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.227+0000 7fbca7eed640 1 -- 172.21.15.16:0/1474525423 >> 172.21.15.16:0/1474525423 conn(0x7fbca00fbd90 msgr2=0x7fbca00fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:50:40.225 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.227+0000 7fbca7eed640 1 -- 172.21.15.16:0/1474525423 shutdown_connections 2024-03-26T04:50:40.225 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.227+0000 7fbca7eed640 1 -- 172.21.15.16:0/1474525423 wait complete. 2024-03-26T04:50:40.227 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.227+0000 7fbca7eed640 1 Processor -- start 2024-03-26T04:50:40.227 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.228+0000 7fbca7eed640 1 -- start start 2024-03-26T04:50:40.227 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.228+0000 7fbca7eed640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbca0119500 con 0x7fbca01025f0 2024-03-26T04:50:40.227 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.228+0000 7fbca7eed640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbca01addc0 con 0x7fbca010a3f0 2024-03-26T04:50:40.228 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.228+0000 7fbca7eed640 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fbca01aef60 con 0x7fbca010df00 2024-03-26T04:50:40.228 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.228+0000 7fbca648a640 1 --1- >> v1:172.21.15.16:6790/0 conn(0x7fbca010df00 0x7fbca0115c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6790/0 says I am v1:172.21.15.16:54500/0 (socket says 172.21.15.16:54500) 2024-03-26T04:50:40.228 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.228+0000 7fbca648a640 1 -- 172.21.15.16:0/3907286407 learned_addr learned my addr 172.21.15.16:0/3907286407 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:50:40.228 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.229+0000 7fbc96ffd640 1 -- 172.21.15.16:0/3907286407 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3822149754 0 0) 0x7fbca01aef60 con 0x7fbca010df00 2024-03-26T04:50:40.228 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.229+0000 7fbc96ffd640 1 -- 172.21.15.16:0/3907286407 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc8c003520 con 0x7fbca010df00 2024-03-26T04:50:40.228 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.229+0000 7fbc96ffd640 1 -- 172.21.15.16:0/3907286407 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1971099904 0 0) 0x7fbca01addc0 con 0x7fbca010a3f0 2024-03-26T04:50:40.228 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.229+0000 7fbc96ffd640 1 -- 172.21.15.16:0/3907286407 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbca01aef60 con 0x7fbca010a3f0 2024-03-26T04:50:40.228 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.229+0000 7fbc96ffd640 1 -- 172.21.15.16:0/3907286407 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2111709863 0 0) 0x7fbca0119500 con 0x7fbca01025f0 2024-03-26T04:50:40.228 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.229+0000 7fbc96ffd640 1 -- 172.21.15.16:0/3907286407 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbca01addc0 con 0x7fbca01025f0 2024-03-26T04:50:40.228 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.229+0000 7fbc96ffd640 1 -- 172.21.15.16:0/3907286407 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1169719715 0 0) 0x7fbc8c003520 con 0x7fbca010df00 2024-03-26T04:50:40.228 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.229+0000 7fbc96ffd640 1 -- 172.21.15.16:0/3907286407 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbca0119500 con 0x7fbca010df00 2024-03-26T04:50:40.228 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.229+0000 7fbc96ffd640 1 -- 172.21.15.16:0/3907286407 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fbc98004730 con 0x7fbca010df00 2024-03-26T04:50:40.229 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.229+0000 7fbc96ffd640 1 -- 172.21.15.16:0/3907286407 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 561215882 0 0) 0x7fbca01aef60 con 0x7fbca010a3f0 2024-03-26T04:50:40.229 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.229+0000 7fbc96ffd640 1 -- 172.21.15.16:0/3907286407 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc8c003520 con 0x7fbca010a3f0 2024-03-26T04:50:40.229 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.229+0000 7fbc96ffd640 1 -- 172.21.15.16:0/3907286407 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fbc900018d0 con 0x7fbca010a3f0 2024-03-26T04:50:40.229 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.229+0000 7fbc96ffd640 1 -- 172.21.15.16:0/3907286407 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4075649777 0 0) 0x7fbca01addc0 con 0x7fbca01025f0 2024-03-26T04:50:40.229 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.229+0000 7fbc96ffd640 1 -- 172.21.15.16:0/3907286407 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbca01aef60 con 0x7fbca01025f0 2024-03-26T04:50:40.229 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.229+0000 7fbc96ffd640 1 -- 172.21.15.16:0/3907286407 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fbc9c001830 con 0x7fbca01025f0 2024-03-26T04:50:40.229 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.229+0000 7fbc96ffd640 1 -- 172.21.15.16:0/3907286407 <== mon.2 v1:172.21.15.16:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2934292233 0 0) 0x7fbca0119500 con 0x7fbca010df00 2024-03-26T04:50:40.229 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.229+0000 7fbc96ffd640 1 -- 172.21.15.16:0/3907286407 >> v1:172.21.15.183:6789/0 conn(0x7fbca010a3f0 legacy=0x7fbca0109b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:50:40.229 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.229+0000 7fbc96ffd640 1 -- 172.21.15.16:0/3907286407 >> v1:172.21.15.16:6789/0 conn(0x7fbca01025f0 legacy=0x7fbca0115510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:50:40.229 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.229+0000 7fbc96ffd640 1 -- 172.21.15.16:0/3907286407 --> v1:172.21.15.16:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca01b0100 con 0x7fbca010df00 2024-03-26T04:50:40.229 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.229+0000 7fbca7eed640 1 -- 172.21.15.16:0/3907286407 --> v1:172.21.15.16:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbca01adfb0 con 0x7fbca010df00 2024-03-26T04:50:40.229 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.229+0000 7fbca7eed640 1 -- 172.21.15.16:0/3907286407 --> v1:172.21.15.16:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbca01ae440 con 0x7fbca010df00 2024-03-26T04:50:40.229 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.229+0000 7fbc96ffd640 1 -- 172.21.15.16:0/3907286407 <== mon.2 v1:172.21.15.16:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7fbc98001e70 con 0x7fbca010df00 2024-03-26T04:50:40.229 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.229+0000 7fbc96ffd640 1 -- 172.21.15.16:0/3907286407 <== mon.2 v1:172.21.15.16:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fbc98005230 con 0x7fbca010df00 2024-03-26T04:50:40.230 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.230+0000 7fbca7eed640 1 -- 172.21.15.16:0/3907286407 --> v1:172.21.15.16:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc84005080 con 0x7fbca010df00 2024-03-26T04:50:40.230 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.231+0000 7fbc96ffd640 1 -- 172.21.15.16:0/3907286407 <== mon.2 v1:172.21.15.16:6790/0 7 ==== mgrmap(e 22) v1 ==== 91420+0+0 (unknown 3095259778 0 0) 0x7fbc98005390 con 0x7fbca010df00 2024-03-26T04:50:40.230 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.231+0000 7fbc96ffd640 1 -- 172.21.15.16:0/3907286407 <== mon.2 v1:172.21.15.16:6790/0 8 ==== osd_map(797..797 src has 253..797) v4 ==== 7785+0+0 (unknown 1079927700 0 0) 0x7fbc98089450 con 0x7fbca010df00 2024-03-26T04:50:40.233 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.236+0000 7fbc96ffd640 1 -- 172.21.15.16:0/3907286407 <== mon.2 v1:172.21.15.16:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fbc98056100 con 0x7fbca010df00 2024-03-26T04:50:40.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:40 smithi016 ceph-mon[29101]: from='client.56813 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "10"}]': finished 2024-03-26T04:50:40.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:40 smithi016 ceph-mon[29101]: osdmap e796: 8 total, 8 up, 8 in 2024-03-26T04:50:40.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:40 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/918547564' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T04:50:40.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:40 smithi016 ceph-mon[29101]: from='client.56813 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T04:50:40.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:40 smithi016 ceph-mon[29101]: pgmap v1713: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:40.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:40 smithi016 ceph-mon[37215]: from='client.56813 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "10"}]': finished 2024-03-26T04:50:40.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:40 smithi016 ceph-mon[37215]: osdmap e796: 8 total, 8 up, 8 in 2024-03-26T04:50:40.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:40 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/918547564' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T04:50:40.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:40 smithi016 ceph-mon[37215]: from='client.56813 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T04:50:40.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:40 smithi016 ceph-mon[37215]: pgmap v1713: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:40.418 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:40.421+0000 7fbca7eed640 1 -- 172.21.15.16:0/3907286407 --> v1:172.21.15.16:6790/0 -- mon_command({"prefix": "osd pool application enable", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "app": "rados"} v 0) v1 -- 0x7fbc84005330 con 0x7fbca010df00 2024-03-26T04:50:40.538 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:40 smithi183 ceph-mon[33745]: from='client.56813 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "10"}]': finished 2024-03-26T04:50:40.539 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:40 smithi183 ceph-mon[33745]: osdmap e796: 8 total, 8 up, 8 in 2024-03-26T04:50:40.539 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:40 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/918547564' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T04:50:40.539 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:40 smithi183 ceph-mon[33745]: from='client.56813 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T04:50:40.539 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:40 smithi183 ceph-mon[33745]: pgmap v1713: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 870 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:50:41.122 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:41.125+0000 7fbc96ffd640 1 -- 172.21.15.16:0/3907286407 <== mon.2 v1:172.21.15.16:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "app": "rados"}]=0 enabled application 'rados' on pool '9afc82fb-6d6b-4f10-901a-c3cc23ea025f' v798) v1 ==== 213+0+0 (unknown 3547132784 0 0) 0x7fbc98054470 con 0x7fbca010df00 2024-03-26T04:50:41.238 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:41.241+0000 7fbca7eed640 1 -- 172.21.15.16:0/3907286407 --> v1:172.21.15.16:6790/0 -- mon_command({"prefix": "osd pool application enable", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "app": "rados"} v 0) v1 -- 0x7fbc84001090 con 0x7fbca010df00 2024-03-26T04:50:41.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:41 smithi016 ceph-mon[29101]: from='client.56813 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "10"}]': finished 2024-03-26T04:50:41.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:41 smithi016 ceph-mon[29101]: osdmap e797: 8 total, 8 up, 8 in 2024-03-26T04:50:41.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:41 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3907286407' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "app": "rados"}]: dispatch 2024-03-26T04:50:41.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:41 smithi016 ceph-mon[29101]: from='client.56819 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "app": "rados"}]: dispatch 2024-03-26T04:50:41.411 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:41 smithi016 ceph-mon[37215]: from='client.56813 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "10"}]': finished 2024-03-26T04:50:41.411 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:41 smithi016 ceph-mon[37215]: osdmap e797: 8 total, 8 up, 8 in 2024-03-26T04:50:41.411 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:41 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3907286407' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "app": "rados"}]: dispatch 2024-03-26T04:50:41.411 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:41 smithi016 ceph-mon[37215]: from='client.56819 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "app": "rados"}]: dispatch 2024-03-26T04:50:41.539 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:41 smithi183 ceph-mon[33745]: from='client.56813 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "10"}]': finished 2024-03-26T04:50:41.539 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:41 smithi183 ceph-mon[33745]: osdmap e797: 8 total, 8 up, 8 in 2024-03-26T04:50:41.539 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:41 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3907286407' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "app": "rados"}]: dispatch 2024-03-26T04:50:41.539 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:41 smithi183 ceph-mon[33745]: from='client.56819 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "app": "rados"}]: dispatch 2024-03-26T04:50:42.136 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:42.138+0000 7fbc96ffd640 1 -- 172.21.15.16:0/3907286407 <== mon.2 v1:172.21.15.16:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "app": "rados"}]=0 enabled application 'rados' on pool '9afc82fb-6d6b-4f10-901a-c3cc23ea025f' v799) v1 ==== 213+0+0 (unknown 3914139030 0 0) 0x7fbc98055c90 con 0x7fbca010df00 2024-03-26T04:50:42.136 INFO:tasks.workunit.client.0.smithi016.stderr:enabled application 'rados' on pool '9afc82fb-6d6b-4f10-901a-c3cc23ea025f' 2024-03-26T04:50:42.139 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:42.141+0000 7fbca7eed640 1 -- 172.21.15.16:0/3907286407 >> v1:172.21.15.16:6800/3868004154 conn(0x7fbc8c06e9e0 legacy=0x7fbc8c070ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:50:42.139 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:42.141+0000 7fbca7eed640 1 -- 172.21.15.16:0/3907286407 >> v1:172.21.15.16:6790/0 conn(0x7fbca010df00 legacy=0x7fbca0115c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:50:42.139 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:42.141+0000 7fbca7eed640 1 -- 172.21.15.16:0/3907286407 shutdown_connections 2024-03-26T04:50:42.139 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:42.141+0000 7fbca7eed640 1 -- 172.21.15.16:0/3907286407 >> 172.21.15.16:0/3907286407 conn(0x7fbca00fbd90 msgr2=0x7fbca01005c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:50:42.139 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:42.141+0000 7fbca7eed640 1 -- 172.21.15.16:0/3907286407 shutdown_connections 2024-03-26T04:50:42.139 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:50:42.141+0000 7fbca7eed640 1 -- 172.21.15.16:0/3907286407 wait complete. 2024-03-26T04:50:42.150 INFO:tasks.workunit.client.0.smithi016.stderr:++ seq 1 10 2024-03-26T04:50:42.170 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in `seq 1 10` 2024-03-26T04:50:42.170 INFO:tasks.workunit.client.0.smithi016.stderr:+ rados -p 9afc82fb-6d6b-4f10-901a-c3cc23ea025f put obj1 /etc/passwd 2024-03-26T04:50:42.206 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in `seq 1 10` 2024-03-26T04:50:42.206 INFO:tasks.workunit.client.0.smithi016.stderr:+ rados -p 9afc82fb-6d6b-4f10-901a-c3cc23ea025f put obj2 /etc/passwd 2024-03-26T04:50:42.243 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in `seq 1 10` 2024-03-26T04:50:42.243 INFO:tasks.workunit.client.0.smithi016.stderr:+ rados -p 9afc82fb-6d6b-4f10-901a-c3cc23ea025f put obj3 /etc/passwd 2024-03-26T04:50:42.277 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in `seq 1 10` 2024-03-26T04:50:42.277 INFO:tasks.workunit.client.0.smithi016.stderr:+ rados -p 9afc82fb-6d6b-4f10-901a-c3cc23ea025f put obj4 /etc/passwd 2024-03-26T04:50:42.309 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in `seq 1 10` 2024-03-26T04:50:42.309 INFO:tasks.workunit.client.0.smithi016.stderr:+ rados -p 9afc82fb-6d6b-4f10-901a-c3cc23ea025f put obj5 /etc/passwd 2024-03-26T04:50:42.344 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in `seq 1 10` 2024-03-26T04:50:42.344 INFO:tasks.workunit.client.0.smithi016.stderr:+ rados -p 9afc82fb-6d6b-4f10-901a-c3cc23ea025f put obj6 /etc/passwd 2024-03-26T04:50:42.376 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in `seq 1 10` 2024-03-26T04:50:42.376 INFO:tasks.workunit.client.0.smithi016.stderr:+ rados -p 9afc82fb-6d6b-4f10-901a-c3cc23ea025f put obj7 /etc/passwd 2024-03-26T04:50:42.409 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in `seq 1 10` 2024-03-26T04:50:42.409 INFO:tasks.workunit.client.0.smithi016.stderr:+ rados -p 9afc82fb-6d6b-4f10-901a-c3cc23ea025f put obj8 /etc/passwd 2024-03-26T04:50:42.409 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:42 smithi016 ceph-mon[37215]: from='client.56819 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "app": "rados"}]': finished 2024-03-26T04:50:42.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:42 smithi016 ceph-mon[37215]: osdmap e798: 8 total, 8 up, 8 in 2024-03-26T04:50:42.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:42 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3907286407' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "app": "rados"}]: dispatch 2024-03-26T04:50:42.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:42 smithi016 ceph-mon[37215]: from='client.56819 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "app": "rados"}]: dispatch 2024-03-26T04:50:42.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:42 smithi016 ceph-mon[37215]: pgmap v1716: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 870 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:50:42.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:42 smithi016 ceph-mon[29101]: from='client.56819 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "app": "rados"}]': finished 2024-03-26T04:50:42.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:42 smithi016 ceph-mon[29101]: osdmap e798: 8 total, 8 up, 8 in 2024-03-26T04:50:42.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:42 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3907286407' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "app": "rados"}]: dispatch 2024-03-26T04:50:42.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:42 smithi016 ceph-mon[29101]: from='client.56819 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "app": "rados"}]: dispatch 2024-03-26T04:50:42.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:42 smithi016 ceph-mon[29101]: pgmap v1716: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 870 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:50:42.441 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in `seq 1 10` 2024-03-26T04:50:42.441 INFO:tasks.workunit.client.0.smithi016.stderr:+ rados -p 9afc82fb-6d6b-4f10-901a-c3cc23ea025f put obj9 /etc/passwd 2024-03-26T04:50:42.474 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in `seq 1 10` 2024-03-26T04:50:42.474 INFO:tasks.workunit.client.0.smithi016.stderr:+ rados -p 9afc82fb-6d6b-4f10-901a-c3cc23ea025f put obj10 /etc/passwd 2024-03-26T04:50:42.506 INFO:tasks.workunit.client.0.smithi016.stderr:+ sleep 30 2024-03-26T04:50:42.538 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:42 smithi183 ceph-mon[33745]: from='client.56819 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "app": "rados"}]': finished 2024-03-26T04:50:42.539 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:42 smithi183 ceph-mon[33745]: osdmap e798: 8 total, 8 up, 8 in 2024-03-26T04:50:42.539 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:42 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3907286407' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "app": "rados"}]: dispatch 2024-03-26T04:50:42.539 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:42 smithi183 ceph-mon[33745]: from='client.56819 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "app": "rados"}]: dispatch 2024-03-26T04:50:42.539 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:42 smithi183 ceph-mon[33745]: pgmap v1716: 176 pgs: 12 unknown, 164 active+clean; 582 KiB data, 870 MiB used, 714 GiB / 715 GiB avail 2024-03-26T04:50:43.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:43 smithi016 ceph-mon[29101]: from='client.56819 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "app": "rados"}]': finished 2024-03-26T04:50:43.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:43 smithi016 ceph-mon[29101]: osdmap e799: 8 total, 8 up, 8 in 2024-03-26T04:50:43.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:43 smithi016 ceph-mon[37215]: from='client.56819 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "app": "rados"}]': finished 2024-03-26T04:50:43.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:43 smithi016 ceph-mon[37215]: osdmap e799: 8 total, 8 up, 8 in 2024-03-26T04:50:43.538 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:43 smithi183 ceph-mon[33745]: from='client.56819 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "app": "rados"}]': finished 2024-03-26T04:50:43.539 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:43 smithi183 ceph-mon[33745]: osdmap e799: 8 total, 8 up, 8 in 2024-03-26T04:50:44.038 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:50:43 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:50:44.409 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:44 smithi016 ceph-mon[37215]: pgmap v1718: 176 pgs: 176 active+clean; 582 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:50:44.410 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:44 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:44.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:44 smithi016 ceph-mon[29101]: pgmap v1718: 176 pgs: 176 active+clean; 582 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:50:44.410 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:44 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:44.538 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:44 smithi183 ceph-mon[33745]: pgmap v1718: 176 pgs: 176 active+clean; 582 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-26T04:50:44.539 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:44 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:47.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:46 smithi183 ceph-mon[33745]: pgmap v1719: 176 pgs: 176 active+clean; 582 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:50:47.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:46 smithi016 ceph-mon[37215]: pgmap v1719: 176 pgs: 176 active+clean; 582 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:50:47.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:46 smithi016 ceph-mon[29101]: pgmap v1719: 176 pgs: 176 active+clean; 582 KiB data, 871 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:50:49.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:48 smithi183 ceph-mon[33745]: pgmap v1720: 176 pgs: 176 active+clean; 698 KiB data, 875 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 16 KiB/s wr, 2 op/s 2024-03-26T04:50:49.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:48 smithi016 ceph-mon[29101]: pgmap v1720: 176 pgs: 176 active+clean; 698 KiB data, 875 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 16 KiB/s wr, 2 op/s 2024-03-26T04:50:49.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:48 smithi016 ceph-mon[37215]: pgmap v1720: 176 pgs: 176 active+clean; 698 KiB data, 875 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 16 KiB/s wr, 2 op/s 2024-03-26T04:50:49.160 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:50:48 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:50:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:50:51.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:50 smithi183 ceph-mon[33745]: pgmap v1721: 176 pgs: 176 active+clean; 698 KiB data, 875 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 14 KiB/s wr, 2 op/s 2024-03-26T04:50:51.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:50 smithi016 ceph-mon[29101]: pgmap v1721: 176 pgs: 176 active+clean; 698 KiB data, 875 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 14 KiB/s wr, 2 op/s 2024-03-26T04:50:51.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:50 smithi016 ceph-mon[37215]: pgmap v1721: 176 pgs: 176 active+clean; 698 KiB data, 875 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 14 KiB/s wr, 2 op/s 2024-03-26T04:50:52.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:51 smithi183 ceph-mon[33745]: pool '9afc82fb-6d6b-4f10-901a-c3cc23ea025f' is full (reached quota's max_objects: 10) 2024-03-26T04:50:52.039 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:51 smithi183 ceph-mon[33745]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T04:50:52.039 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:51 smithi183 ceph-mon[33745]: osdmap e800: 8 total, 8 up, 8 in 2024-03-26T04:50:52.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:51 smithi016 ceph-mon[37215]: pool '9afc82fb-6d6b-4f10-901a-c3cc23ea025f' is full (reached quota's max_objects: 10) 2024-03-26T04:50:52.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:51 smithi016 ceph-mon[37215]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T04:50:52.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:51 smithi016 ceph-mon[37215]: osdmap e800: 8 total, 8 up, 8 in 2024-03-26T04:50:52.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:51 smithi016 ceph-mon[29101]: pool '9afc82fb-6d6b-4f10-901a-c3cc23ea025f' is full (reached quota's max_objects: 10) 2024-03-26T04:50:52.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:51 smithi016 ceph-mon[29101]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T04:50:52.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:51 smithi016 ceph-mon[29101]: osdmap e800: 8 total, 8 up, 8 in 2024-03-26T04:50:53.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:52 smithi183 ceph-mon[33745]: pgmap v1723: 176 pgs: 176 active+clean; 698 KiB data, 875 MiB used, 714 GiB / 715 GiB avail; 538 B/s rd, 13 KiB/s wr, 1 op/s 2024-03-26T04:50:53.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:52 smithi016 ceph-mon[37215]: pgmap v1723: 176 pgs: 176 active+clean; 698 KiB data, 875 MiB used, 714 GiB / 715 GiB avail; 538 B/s rd, 13 KiB/s wr, 1 op/s 2024-03-26T04:50:53.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:52 smithi016 ceph-mon[29101]: pgmap v1723: 176 pgs: 176 active+clean; 698 KiB data, 875 MiB used, 714 GiB / 715 GiB avail; 538 B/s rd, 13 KiB/s wr, 1 op/s 2024-03-26T04:50:54.038 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:50:53 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:50:55.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:54 smithi016 ceph-mon[29101]: pgmap v1724: 176 pgs: 176 active+clean; 698 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T04:50:55.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:54 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:55.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:54 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:50:55.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:54 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:50:55.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:54 smithi016 ceph-mon[37215]: pgmap v1724: 176 pgs: 176 active+clean; 698 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T04:50:55.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:54 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:55.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:54 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:50:55.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:54 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:50:55.288 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:54 smithi183 ceph-mon[33745]: pgmap v1724: 176 pgs: 176 active+clean; 698 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T04:50:55.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:54 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:50:55.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:54 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:50:55.289 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:54 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:50:57.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:56 smithi016 ceph-mon[29101]: pgmap v1725: 176 pgs: 176 active+clean; 698 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T04:50:57.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:56 smithi016 ceph-mon[37215]: pgmap v1725: 176 pgs: 176 active+clean; 698 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T04:50:57.288 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:56 smithi183 ceph-mon[33745]: pgmap v1725: 176 pgs: 176 active+clean; 698 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T04:50:59.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:50:58 smithi016 ceph-mon[29101]: pgmap v1726: 176 pgs: 176 active+clean; 698 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:50:59.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:50:58 smithi016 ceph-mon[37215]: pgmap v1726: 176 pgs: 176 active+clean; 698 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:50:59.160 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:50:58 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:50:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:50:59.288 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:50:58 smithi183 ceph-mon[33745]: pgmap v1726: 176 pgs: 176 active+clean; 698 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:51:01.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:00 smithi016 ceph-mon[37215]: pgmap v1727: 176 pgs: 176 active+clean; 698 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:51:01.161 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:00 smithi016 ceph-mon[29101]: pgmap v1727: 176 pgs: 176 active+clean; 698 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:51:01.288 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:00 smithi183 ceph-mon[33745]: pgmap v1727: 176 pgs: 176 active+clean; 698 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:51:02.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:01 smithi016 ceph-mon[37215]: pgmap v1728: 176 pgs: 176 active+clean; 698 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1000 B/s rd, 0 op/s 2024-03-26T04:51:02.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:01 smithi016 ceph-mon[29101]: pgmap v1728: 176 pgs: 176 active+clean; 698 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1000 B/s rd, 0 op/s 2024-03-26T04:51:02.288 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:01 smithi183 ceph-mon[33745]: pgmap v1728: 176 pgs: 176 active+clean; 698 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1000 B/s rd, 0 op/s 2024-03-26T04:51:04.038 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:51:03 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:51:05.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:04 smithi183 ceph-mon[33745]: pgmap v1729: 176 pgs: 176 active+clean; 698 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:51:05.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:04 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:51:05.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:04 smithi016 ceph-mon[29101]: pgmap v1729: 176 pgs: 176 active+clean; 698 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:51:05.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:04 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:51:05.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:04 smithi016 ceph-mon[37215]: pgmap v1729: 176 pgs: 176 active+clean; 698 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:51:05.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:04 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:51:07.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:06 smithi183 ceph-mon[33745]: pgmap v1730: 176 pgs: 176 active+clean; 698 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:51:07.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:06 smithi016 ceph-mon[29101]: pgmap v1730: 176 pgs: 176 active+clean; 698 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:51:07.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:06 smithi016 ceph-mon[37215]: pgmap v1730: 176 pgs: 176 active+clean; 698 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:51:09.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:08 smithi183 ceph-mon[33745]: pgmap v1731: 176 pgs: 176 active+clean; 698 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:51:09.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:08 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:51:09.159 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:51:08 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:51:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:51:09.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:08 smithi016 ceph-mon[37215]: pgmap v1731: 176 pgs: 176 active+clean; 698 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:51:09.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:08 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:51:09.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:08 smithi016 ceph-mon[29101]: pgmap v1731: 176 pgs: 176 active+clean; 698 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:51:09.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:08 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:51:10.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:09 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:51:10.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:09 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:51:10.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:09 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:51:11.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:10 smithi183 ceph-mon[33745]: pgmap v1732: 176 pgs: 176 active+clean; 698 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:51:11.132 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:10 smithi016 ceph-mon[37215]: pgmap v1732: 176 pgs: 176 active+clean; 698 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:51:11.133 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:10 smithi016 ceph-mon[29101]: pgmap v1732: 176 pgs: 176 active+clean; 698 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:51:12.507 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=150918 2024-03-26T04:51:12.508 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph osd pool set-quota 9afc82fb-6d6b-4f10-901a-c3cc23ea025f max_objects 100 2024-03-26T04:51:12.508 INFO:tasks.workunit.client.0.smithi016.stderr:+ rados -p 9afc82fb-6d6b-4f10-901a-c3cc23ea025f put onemore /etc/passwd 2024-03-26T04:51:12.594 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:12.596+0000 7fdcfd46b640 1 -- 172.21.15.16:0/1961250291 >> v1:172.21.15.16:6790/0 conn(0x7fdcf8107490 legacy=0x7fdcf8107870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:51:12.594 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:12.596+0000 7fdcfd46b640 1 -- 172.21.15.16:0/1961250291 shutdown_connections 2024-03-26T04:51:12.594 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:12.596+0000 7fdcfd46b640 1 -- 172.21.15.16:0/1961250291 >> 172.21.15.16:0/1961250291 conn(0x7fdcf8075610 msgr2=0x7fdcf8075a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:51:12.594 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:12.596+0000 7fdcfd46b640 1 -- 172.21.15.16:0/1961250291 shutdown_connections 2024-03-26T04:51:12.594 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:12.596+0000 7fdcfd46b640 1 -- 172.21.15.16:0/1961250291 wait complete. 2024-03-26T04:51:12.595 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:12.596+0000 7fdcfd46b640 1 Processor -- start 2024-03-26T04:51:12.595 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:12.597+0000 7fdcfd46b640 1 -- start start 2024-03-26T04:51:12.595 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:12.597+0000 7fdcfd46b640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdcf81a8590 con 0x7fdcf810de70 2024-03-26T04:51:12.595 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:12.597+0000 7fdcfd46b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdcf81a9790 con 0x7fdcf8107490 2024-03-26T04:51:12.596 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:12.597+0000 7fdcfd46b640 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fdcf81aa990 con 0x7fdcf810a2a0 2024-03-26T04:51:12.596 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:12.597+0000 7fdcf6ffd640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fdcf8107490 0x7fdcf8100390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.16:46954/0 (socket says 172.21.15.16:46954) 2024-03-26T04:51:12.596 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:12.597+0000 7fdcf6ffd640 1 -- 172.21.15.16:0/3130350770 learned_addr learned my addr 172.21.15.16:0/3130350770 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:51:12.601 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:12.600+0000 7fdceffff640 1 -- 172.21.15.16:0/3130350770 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1051029437 0 0) 0x7fdcf81a8590 con 0x7fdcf810de70 2024-03-26T04:51:12.601 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:12.600+0000 7fdceffff640 1 -- 172.21.15.16:0/3130350770 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdce4003520 con 0x7fdcf810de70 2024-03-26T04:51:12.601 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:12.600+0000 7fdceffff640 1 -- 172.21.15.16:0/3130350770 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1511082756 0 0) 0x7fdcf81a9790 con 0x7fdcf8107490 2024-03-26T04:51:12.601 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:12.600+0000 7fdceffff640 1 -- 172.21.15.16:0/3130350770 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcf81a8590 con 0x7fdcf8107490 2024-03-26T04:51:12.601 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:12.600+0000 7fdceffff640 1 -- 172.21.15.16:0/3130350770 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2077724349 0 0) 0x7fdcf81aa990 con 0x7fdcf810a2a0 2024-03-26T04:51:12.602 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:12.600+0000 7fdceffff640 1 -- 172.21.15.16:0/3130350770 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcf81a9790 con 0x7fdcf810a2a0 2024-03-26T04:51:12.602 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:12.600+0000 7fdceffff640 1 -- 172.21.15.16:0/3130350770 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2294543265 0 0) 0x7fdcf81a8590 con 0x7fdcf8107490 2024-03-26T04:51:12.602 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:12.600+0000 7fdceffff640 1 -- 172.21.15.16:0/3130350770 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdcf81aa990 con 0x7fdcf8107490 2024-03-26T04:51:12.602 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:12.600+0000 7fdceffff640 1 -- 172.21.15.16:0/3130350770 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3068963696 0 0) 0x7fdcf81a9790 con 0x7fdcf810a2a0 2024-03-26T04:51:12.602 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:12.600+0000 7fdceffff640 1 -- 172.21.15.16:0/3130350770 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdcf81a8590 con 0x7fdcf810a2a0 2024-03-26T04:51:12.602 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:12.600+0000 7fdceffff640 1 -- 172.21.15.16:0/3130350770 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3774858569 0 0) 0x7fdce4003520 con 0x7fdcf810de70 2024-03-26T04:51:12.602 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:12.600+0000 7fdceffff640 1 -- 172.21.15.16:0/3130350770 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdcf81a9790 con 0x7fdcf810de70 2024-03-26T04:51:12.602 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:12.600+0000 7fdceffff640 1 -- 172.21.15.16:0/3130350770 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fdce00015d0 con 0x7fdcf8107490 2024-03-26T04:51:12.602 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:12.600+0000 7fdceffff640 1 -- 172.21.15.16:0/3130350770 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fdce80017b0 con 0x7fdcf810a2a0 2024-03-26T04:51:12.602 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:12.600+0000 7fdceffff640 1 -- 172.21.15.16:0/3130350770 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fdcf00013f0 con 0x7fdcf810de70 2024-03-26T04:51:12.602 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:12.600+0000 7fdceffff640 1 -- 172.21.15.16:0/3130350770 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3619043619 0 0) 0x7fdcf81aa990 con 0x7fdcf8107490 2024-03-26T04:51:12.602 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:12.600+0000 7fdceffff640 1 -- 172.21.15.16:0/3130350770 >> v1:172.21.15.16:6790/0 conn(0x7fdcf810a2a0 legacy=0x7fdcf81a35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:51:12.602 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:12.600+0000 7fdceffff640 1 -- 172.21.15.16:0/3130350770 >> v1:172.21.15.16:6789/0 conn(0x7fdcf810de70 legacy=0x7fdcf81a6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:51:12.602 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:12.600+0000 7fdceffff640 1 -- 172.21.15.16:0/3130350770 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcf81abb90 con 0x7fdcf8107490 2024-03-26T04:51:12.603 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:12.600+0000 7fdcfd46b640 1 -- 172.21.15.16:0/3130350770 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcf81a8780 con 0x7fdcf8107490 2024-03-26T04:51:12.603 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:12.600+0000 7fdcfd46b640 1 -- 172.21.15.16:0/3130350770 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcf81a8c10 con 0x7fdcf8107490 2024-03-26T04:51:12.603 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:12.600+0000 7fdceffff640 1 -- 172.21.15.16:0/3130350770 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7fdce0003a80 con 0x7fdcf8107490 2024-03-26T04:51:12.603 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:12.600+0000 7fdceffff640 1 -- 172.21.15.16:0/3130350770 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fdce0004d80 con 0x7fdcf8107490 2024-03-26T04:51:12.603 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:12.601+0000 7fdcfd46b640 1 -- 172.21.15.16:0/3130350770 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcd8005080 con 0x7fdcf8107490 2024-03-26T04:51:12.603 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:12.602+0000 7fdceffff640 1 -- 172.21.15.16:0/3130350770 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 22) v1 ==== 91420+0+0 (unknown 3095259778 0 0) 0x7fdce0004ee0 con 0x7fdcf8107490 2024-03-26T04:51:12.603 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:12.602+0000 7fdceffff640 1 -- 172.21.15.16:0/3130350770 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(800..800 src has 253..800) v4 ==== 7798+0+0 (unknown 157552215 0 0) 0x7fdce0089180 con 0x7fdcf8107490 2024-03-26T04:51:12.603 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:12.602+0000 7fdceffff640 1 -- 172.21.15.16:0/3130350770 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=801}) v3 -- 0x7fdcf81aa990 con 0x7fdcf8107490 2024-03-26T04:51:12.605 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:12.607+0000 7fdceffff640 1 -- 172.21.15.16:0/3130350770 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fdce0055e20 con 0x7fdcf8107490 2024-03-26T04:51:12.692 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:12 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:51:12.693 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:12 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:51:12.693 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:12 smithi183 ceph-mon[33745]: pgmap v1733: 176 pgs: 176 active+clean; 698 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:51:12.788 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:12.790+0000 7fdcfd46b640 1 -- 172.21.15.16:0/3130350770 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "100"} v 0) v1 -- 0x7fdcd8005330 con 0x7fdcf8107490 2024-03-26T04:51:12.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:12 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:51:12.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:12 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:51:12.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:12 smithi016 ceph-mon[29101]: pgmap v1733: 176 pgs: 176 active+clean; 698 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:51:12.909 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:12 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:51:12.909 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:12 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:51:12.909 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:12 smithi016 ceph-mon[37215]: pgmap v1733: 176 pgs: 176 active+clean; 698 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:51:13.617 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:13.619+0000 7fdceffff640 1 -- 172.21.15.16:0/3130350770 <== mon.1 v1:172.21.15.183:6789/0 10 ==== osd_map(801..801 src has 253..801) v4 ==== 628+0+0 (unknown 3451867387 0 0) 0x7fdce0056750 con 0x7fdcf8107490 2024-03-26T04:51:13.617 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:13.619+0000 7fdceffff640 1 -- 172.21.15.16:0/3130350770 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=802}) v3 -- 0x7fdcf81a9790 con 0x7fdcf8107490 2024-03-26T04:51:13.621 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:13.624+0000 7fdceffff640 1 -- 172.21.15.16:0/3130350770 <== mon.1 v1:172.21.15.183:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "100"}]=0 set-quota max_objects = 100 for pool 9afc82fb-6d6b-4f10-901a-c3cc23ea025f v801) v1 ==== 225+0+0 (unknown 3472285220 0 0) 0x7fdce0054190 con 0x7fdcf8107490 2024-03-26T04:51:13.732 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:13.734+0000 7fdcfd46b640 1 -- 172.21.15.16:0/3130350770 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "100"} v 0) v1 -- 0x7fdcd80010c0 con 0x7fdcf8107490 2024-03-26T04:51:13.909 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:13 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3130350770' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T04:51:13.909 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:13 smithi016 ceph-mon[37215]: from='client.49438 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T04:51:13.909 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:13 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:51:13.909 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:13 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:51:13.909 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:13 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:51:13.909 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:13 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:51:13.909 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:13 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:51:13.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:13 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3130350770' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T04:51:13.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:13 smithi016 ceph-mon[29101]: from='client.49438 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T04:51:13.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:13 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:51:13.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:13 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:51:13.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:13 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:51:13.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:13 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:51:13.911 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:13 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:51:14.038 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:51:13 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:51:14.039 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:13 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3130350770' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T04:51:14.039 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:13 smithi183 ceph-mon[33745]: from='client.49438 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T04:51:14.039 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:13 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:51:14.039 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:13 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:51:14.039 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:13 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:51:14.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:13 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:51:14.040 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:13 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:51:14.620 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:14.622+0000 7fdceffff640 1 -- 172.21.15.16:0/3130350770 <== mon.1 v1:172.21.15.183:6789/0 12 ==== osd_map(802..802 src has 253..802) v4 ==== 628+0+0 (unknown 3577981496 0 0) 0x7fdce005a1a0 con 0x7fdcf8107490 2024-03-26T04:51:14.620 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:14.622+0000 7fdceffff640 1 -- 172.21.15.16:0/3130350770 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=803}) v3 -- 0x7fdcf81a8590 con 0x7fdcf8107490 2024-03-26T04:51:14.625 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:14.627+0000 7fdceffff640 1 -- 172.21.15.16:0/3130350770 <== mon.1 v1:172.21.15.183:6789/0 13 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "100"}]=0 set-quota max_objects = 100 for pool 9afc82fb-6d6b-4f10-901a-c3cc23ea025f v802) v1 ==== 225+0+0 (unknown 3084098220 0 0) 0x7fdce00559b0 con 0x7fdcf8107490 2024-03-26T04:51:14.625 INFO:tasks.workunit.client.0.smithi016.stderr:set-quota max_objects = 100 for pool 9afc82fb-6d6b-4f10-901a-c3cc23ea025f 2024-03-26T04:51:14.629 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:14.630+0000 7fdcfd46b640 1 -- 172.21.15.16:0/3130350770 >> v1:172.21.15.16:6800/3868004154 conn(0x7fdce406ea10 legacy=0x7fdce4070ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:51:14.629 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:14.630+0000 7fdcfd46b640 1 -- 172.21.15.16:0/3130350770 >> v1:172.21.15.183:6789/0 conn(0x7fdcf8107490 legacy=0x7fdcf8100390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:51:14.629 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:14.630+0000 7fdcfd46b640 1 -- 172.21.15.16:0/3130350770 shutdown_connections 2024-03-26T04:51:14.629 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:14.630+0000 7fdcfd46b640 1 -- 172.21.15.16:0/3130350770 >> 172.21.15.16:0/3130350770 conn(0x7fdcf8075610 msgr2=0x7fdcf81102d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:51:14.629 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:14.631+0000 7fdcfd46b640 1 -- 172.21.15.16:0/3130350770 shutdown_connections 2024-03-26T04:51:14.629 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:14.631+0000 7fdcfd46b640 1 -- 172.21.15.16:0/3130350770 wait complete. 2024-03-26T04:51:14.640 INFO:tasks.workunit.client.0.smithi016.stderr:+ wait 150918 2024-03-26T04:51:14.910 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:14 smithi016 ceph-mon[37215]: from='client.49438 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "100"}]': finished 2024-03-26T04:51:14.910 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:14 smithi016 ceph-mon[37215]: osdmap e801: 8 total, 8 up, 8 in 2024-03-26T04:51:14.910 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:14 smithi016 ceph-mon[37215]: pgmap v1735: 176 pgs: 176 active+clean; 698 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:51:14.910 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:14 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:51:14.910 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:14 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3130350770' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T04:51:14.910 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:14 smithi016 ceph-mon[37215]: from='client.49438 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T04:51:14.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:14 smithi016 ceph-mon[29101]: from='client.49438 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "100"}]': finished 2024-03-26T04:51:14.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:14 smithi016 ceph-mon[29101]: osdmap e801: 8 total, 8 up, 8 in 2024-03-26T04:51:14.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:14 smithi016 ceph-mon[29101]: pgmap v1735: 176 pgs: 176 active+clean; 698 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:51:14.911 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:14 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:51:14.911 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:14 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3130350770' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T04:51:14.911 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:14 smithi016 ceph-mon[29101]: from='client.49438 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T04:51:15.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:14 smithi183 ceph-mon[33745]: from='client.49438 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "100"}]': finished 2024-03-26T04:51:15.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:14 smithi183 ceph-mon[33745]: osdmap e801: 8 total, 8 up, 8 in 2024-03-26T04:51:15.039 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:14 smithi183 ceph-mon[33745]: pgmap v1735: 176 pgs: 176 active+clean; 698 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:51:15.039 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:14 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:51:15.039 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:14 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3130350770' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T04:51:15.039 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:14 smithi183 ceph-mon[33745]: from='client.49438 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "100"}]: dispatch 2024-03-26T04:51:15.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:15 smithi016 ceph-mon[29101]: from='client.49438 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "100"}]': finished 2024-03-26T04:51:15.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:15 smithi016 ceph-mon[29101]: osdmap e802: 8 total, 8 up, 8 in 2024-03-26T04:51:15.909 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:15 smithi016 ceph-mon[37215]: from='client.49438 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "100"}]': finished 2024-03-26T04:51:15.909 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:15 smithi016 ceph-mon[37215]: osdmap e802: 8 total, 8 up, 8 in 2024-03-26T04:51:16.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:15 smithi183 ceph-mon[33745]: from='client.49438 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "100"}]': finished 2024-03-26T04:51:16.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:15 smithi183 ceph-mon[33745]: osdmap e802: 8 total, 8 up, 8 in 2024-03-26T04:51:16.423 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' 0 -ne 0 ']' 2024-03-26T04:51:16.424 INFO:tasks.workunit.client.0.smithi016.stderr:+ true 2024-03-26T04:51:16.424 INFO:tasks.workunit.client.0.smithi016.stderr:+ rados -p 9afc82fb-6d6b-4f10-901a-c3cc23ea025f put twomore /etc/passwd 2024-03-26T04:51:16.461 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph osd pool set-quota 9afc82fb-6d6b-4f10-901a-c3cc23ea025f max_bytes 100 2024-03-26T04:51:16.551 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:16.550+0000 7f2a0b047640 1 -- 172.21.15.16:0/2967469683 >> v1:172.21.15.16:6790/0 conn(0x7f2a041091c0 legacy=0x7f2a0410b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:51:16.551 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:16.550+0000 7f2a0b047640 1 -- 172.21.15.16:0/2967469683 shutdown_connections 2024-03-26T04:51:16.551 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:16.550+0000 7f2a0b047640 1 -- 172.21.15.16:0/2967469683 >> 172.21.15.16:0/2967469683 conn(0x7f2a04076fa0 msgr2=0x7f2a040fe300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:51:16.551 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:16.550+0000 7f2a0b047640 1 -- 172.21.15.16:0/2967469683 shutdown_connections 2024-03-26T04:51:16.551 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:16.550+0000 7f2a0b047640 1 -- 172.21.15.16:0/2967469683 wait complete. 2024-03-26T04:51:16.551 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:16.551+0000 7f2a0b047640 1 Processor -- start 2024-03-26T04:51:16.552 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:16.551+0000 7f2a0b047640 1 -- start start 2024-03-26T04:51:16.552 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:16.551+0000 7f2a0b047640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2a041a5200 con 0x7f2a0410cd30 2024-03-26T04:51:16.552 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:16.551+0000 7f2a0b047640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2a041a63c0 con 0x7f2a04100540 2024-03-26T04:51:16.552 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:16.551+0000 7f2a0b047640 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f2a041a75c0 con 0x7f2a041091c0 2024-03-26T04:51:16.552 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:16.551+0000 7f2a03fff640 1 --1- >> v1:172.21.15.16:6790/0 conn(0x7f2a041091c0 0x7f2a041a00a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6790/0 says I am v1:172.21.15.16:50086/0 (socket says 172.21.15.16:50086) 2024-03-26T04:51:16.552 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:16.551+0000 7f2a03fff640 1 -- 172.21.15.16:0/2944532725 learned_addr learned my addr 172.21.15.16:0/2944532725 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:51:16.552 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:16.551+0000 7f2a01ffb640 1 -- 172.21.15.16:0/2944532725 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 187435407 0 0) 0x7f2a041a5200 con 0x7f2a0410cd30 2024-03-26T04:51:16.552 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:16.551+0000 7f2a01ffb640 1 -- 172.21.15.16:0/2944532725 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29f4003520 con 0x7f2a0410cd30 2024-03-26T04:51:16.552 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:16.551+0000 7f2a01ffb640 1 -- 172.21.15.16:0/2944532725 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3404719284 0 0) 0x7f2a041a75c0 con 0x7f2a041091c0 2024-03-26T04:51:16.552 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:16.551+0000 7f2a01ffb640 1 -- 172.21.15.16:0/2944532725 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a041a5200 con 0x7f2a041091c0 2024-03-26T04:51:16.552 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:16.551+0000 7f2a01ffb640 1 -- 172.21.15.16:0/2944532725 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1410412034 0 0) 0x7f2a041a63c0 con 0x7f2a04100540 2024-03-26T04:51:16.552 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:16.551+0000 7f2a01ffb640 1 -- 172.21.15.16:0/2944532725 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a041a75c0 con 0x7f2a04100540 2024-03-26T04:51:16.552 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:16.551+0000 7f2a01ffb640 1 -- 172.21.15.16:0/2944532725 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3341798981 0 0) 0x7f29f4003520 con 0x7f2a0410cd30 2024-03-26T04:51:16.552 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:16.551+0000 7f2a01ffb640 1 -- 172.21.15.16:0/2944532725 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a041a63c0 con 0x7f2a0410cd30 2024-03-26T04:51:16.553 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:16.551+0000 7f2a01ffb640 1 -- 172.21.15.16:0/2944532725 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f29fc001b50 con 0x7f2a0410cd30 2024-03-26T04:51:16.553 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:16.551+0000 7f2a01ffb640 1 -- 172.21.15.16:0/2944532725 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 326399222 0 0) 0x7f2a041a5200 con 0x7f2a041091c0 2024-03-26T04:51:16.553 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:16.552+0000 7f2a01ffb640 1 -- 172.21.15.16:0/2944532725 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29f4003520 con 0x7f2a041091c0 2024-03-26T04:51:16.553 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:16.552+0000 7f2a01ffb640 1 -- 172.21.15.16:0/2944532725 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f29f0004480 con 0x7f2a041091c0 2024-03-26T04:51:16.553 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:16.552+0000 7f2a01ffb640 1 -- 172.21.15.16:0/2944532725 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2062249347 0 0) 0x7f2a041a75c0 con 0x7f2a04100540 2024-03-26T04:51:16.553 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:16.552+0000 7f2a01ffb640 1 -- 172.21.15.16:0/2944532725 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a041a5200 con 0x7f2a04100540 2024-03-26T04:51:16.553 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:16.552+0000 7f2a01ffb640 1 -- 172.21.15.16:0/2944532725 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f29f8001830 con 0x7f2a04100540 2024-03-26T04:51:16.553 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:16.552+0000 7f2a01ffb640 1 -- 172.21.15.16:0/2944532725 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4202406186 0 0) 0x7f2a041a63c0 con 0x7f2a0410cd30 2024-03-26T04:51:16.553 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:16.552+0000 7f2a01ffb640 1 -- 172.21.15.16:0/2944532725 >> v1:172.21.15.16:6790/0 conn(0x7f2a041091c0 legacy=0x7f2a041a00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:51:16.553 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:16.552+0000 7f2a01ffb640 1 -- 172.21.15.16:0/2944532725 >> v1:172.21.15.183:6789/0 conn(0x7f2a04100540 legacy=0x7f2a041057e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:51:16.553 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:16.552+0000 7f2a01ffb640 1 -- 172.21.15.16:0/2944532725 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a041a87c0 con 0x7f2a0410cd30 2024-03-26T04:51:16.553 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:16.552+0000 7f2a0b047640 1 -- 172.21.15.16:0/2944532725 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a041a53f0 con 0x7f2a0410cd30 2024-03-26T04:51:16.553 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:16.552+0000 7f2a01ffb640 1 -- 172.21.15.16:0/2944532725 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f29fc002080 con 0x7f2a0410cd30 2024-03-26T04:51:16.553 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:16.552+0000 7f2a01ffb640 1 -- 172.21.15.16:0/2944532725 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f29fc004c20 con 0x7f2a0410cd30 2024-03-26T04:51:16.553 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:16.552+0000 7f2a0b047640 1 -- 172.21.15.16:0/2944532725 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a041a5880 con 0x7f2a0410cd30 2024-03-26T04:51:16.554 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:16.553+0000 7f2a0b047640 1 -- 172.21.15.16:0/2944532725 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a04101a10 con 0x7f2a0410cd30 2024-03-26T04:51:16.554 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:16.553+0000 7f2a01ffb640 1 -- 172.21.15.16:0/2944532725 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 22) v1 ==== 91420+0+0 (unknown 3095259778 0 0) 0x7f29fc003970 con 0x7f2a0410cd30 2024-03-26T04:51:16.554 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:16.554+0000 7f2a01ffb640 1 -- 172.21.15.16:0/2944532725 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(803..803 src has 253..803) v4 ==== 7798+0+0 (unknown 876682597 0 0) 0x7f29fc087f10 con 0x7f2a0410cd30 2024-03-26T04:51:16.556 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:16.558+0000 7f2a01ffb640 1 -- 172.21.15.16:0/2944532725 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f29fc054c10 con 0x7f2a0410cd30 2024-03-26T04:51:16.745 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:16.747+0000 7f2a0b047640 1 -- 172.21.15.16:0/2944532725 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "100"} v 0) v1 -- 0x7f2a04068210 con 0x7f2a0410cd30 2024-03-26T04:51:16.909 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:16 smithi016 ceph-mon[37215]: pgmap v1737: 176 pgs: 176 active+clean; 698 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:51:16.909 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:16 smithi016 ceph-mon[37215]: pool '9afc82fb-6d6b-4f10-901a-c3cc23ea025f' no longer out of quota; removing NO_QUOTA flag 2024-03-26T04:51:16.909 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:16 smithi016 ceph-mon[37215]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T04:51:16.909 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:16 smithi016 ceph-mon[37215]: osdmap e803: 8 total, 8 up, 8 in 2024-03-26T04:51:16.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:16 smithi016 ceph-mon[29101]: pgmap v1737: 176 pgs: 176 active+clean; 698 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:51:16.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:16 smithi016 ceph-mon[29101]: pool '9afc82fb-6d6b-4f10-901a-c3cc23ea025f' no longer out of quota; removing NO_QUOTA flag 2024-03-26T04:51:16.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:16 smithi016 ceph-mon[29101]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T04:51:16.910 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:16 smithi016 ceph-mon[29101]: osdmap e803: 8 total, 8 up, 8 in 2024-03-26T04:51:17.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:16 smithi183 ceph-mon[33745]: pgmap v1737: 176 pgs: 176 active+clean; 698 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:51:17.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:16 smithi183 ceph-mon[33745]: pool '9afc82fb-6d6b-4f10-901a-c3cc23ea025f' no longer out of quota; removing NO_QUOTA flag 2024-03-26T04:51:17.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:16 smithi183 ceph-mon[33745]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T04:51:17.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:16 smithi183 ceph-mon[33745]: osdmap e803: 8 total, 8 up, 8 in 2024-03-26T04:51:17.631 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:17.633+0000 7f2a01ffb640 1 -- 172.21.15.16:0/2944532725 <== mon.0 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "100"}]=0 set-quota max_bytes = 100 for pool 9afc82fb-6d6b-4f10-901a-c3cc23ea025f v804) v1 ==== 221+0+0 (unknown 3185314881 0 0) 0x7f29fc052f80 con 0x7f2a0410cd30 2024-03-26T04:51:17.749 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:17.751+0000 7f2a0b047640 1 -- 172.21.15.16:0/2944532725 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "100"} v 0) v1 -- 0x7f2a04103db0 con 0x7f2a0410cd30 2024-03-26T04:51:17.909 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:17 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2944532725' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T04:51:17.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:17 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2944532725' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T04:51:18.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:17 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2944532725' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T04:51:18.636 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:18.638+0000 7f2a01ffb640 1 -- 172.21.15.16:0/2944532725 <== mon.0 v1:172.21.15.16:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "100"}]=0 set-quota max_bytes = 100 for pool 9afc82fb-6d6b-4f10-901a-c3cc23ea025f v805) v1 ==== 221+0+0 (unknown 1032641952 0 0) 0x7f29fc0547a0 con 0x7f2a0410cd30 2024-03-26T04:51:18.637 INFO:tasks.workunit.client.0.smithi016.stderr:set-quota max_bytes = 100 for pool 9afc82fb-6d6b-4f10-901a-c3cc23ea025f 2024-03-26T04:51:18.641 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:18.642+0000 7f2a0b047640 1 -- 172.21.15.16:0/2944532725 >> v1:172.21.15.16:6800/3868004154 conn(0x7f29f4070d00 legacy=0x7f29f40731c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:51:18.641 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:18.642+0000 7f2a0b047640 1 -- 172.21.15.16:0/2944532725 >> v1:172.21.15.16:6789/0 conn(0x7f2a0410cd30 legacy=0x7f2a041a38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:51:18.641 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:18.642+0000 7f2a0b047640 1 -- 172.21.15.16:0/2944532725 shutdown_connections 2024-03-26T04:51:18.641 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:18.642+0000 7f2a0b047640 1 -- 172.21.15.16:0/2944532725 >> 172.21.15.16:0/2944532725 conn(0x7f2a04076fa0 msgr2=0x7f2a0410da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:51:18.641 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:18.643+0000 7f2a0b047640 1 -- 172.21.15.16:0/2944532725 shutdown_connections 2024-03-26T04:51:18.641 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:18.643+0000 7f2a0b047640 1 -- 172.21.15.16:0/2944532725 wait complete. 2024-03-26T04:51:18.653 INFO:tasks.workunit.client.0.smithi016.stderr:+ sleep 30 2024-03-26T04:51:18.909 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:51:18 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:51:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:51:18.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:18 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2944532725' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "100"}]': finished 2024-03-26T04:51:18.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:18 smithi016 ceph-mon[29101]: osdmap e804: 8 total, 8 up, 8 in 2024-03-26T04:51:18.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:18 smithi016 ceph-mon[29101]: pgmap v1740: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-03-26T04:51:18.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:18 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2944532725' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T04:51:18.910 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:18 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2944532725' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "100"}]': finished 2024-03-26T04:51:18.910 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:18 smithi016 ceph-mon[37215]: osdmap e804: 8 total, 8 up, 8 in 2024-03-26T04:51:18.910 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:18 smithi016 ceph-mon[37215]: pgmap v1740: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-03-26T04:51:18.910 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:18 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2944532725' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T04:51:19.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:18 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2944532725' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "100"}]': finished 2024-03-26T04:51:19.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:18 smithi183 ceph-mon[33745]: osdmap e804: 8 total, 8 up, 8 in 2024-03-26T04:51:19.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:18 smithi183 ceph-mon[33745]: pgmap v1740: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 6.0 KiB/s wr, 1 op/s 2024-03-26T04:51:19.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:18 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2944532725' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "100"}]: dispatch 2024-03-26T04:51:19.909 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:19 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/2944532725' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "100"}]': finished 2024-03-26T04:51:19.909 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:19 smithi016 ceph-mon[37215]: osdmap e805: 8 total, 8 up, 8 in 2024-03-26T04:51:19.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:19 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/2944532725' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "100"}]': finished 2024-03-26T04:51:19.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:19 smithi016 ceph-mon[29101]: osdmap e805: 8 total, 8 up, 8 in 2024-03-26T04:51:20.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:19 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/2944532725' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "100"}]': finished 2024-03-26T04:51:20.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:19 smithi183 ceph-mon[33745]: osdmap e805: 8 total, 8 up, 8 in 2024-03-26T04:51:20.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:20 smithi016 ceph-mon[29101]: pgmap v1742: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1017 B/s rd, 4.8 KiB/s wr, 1 op/s 2024-03-26T04:51:20.909 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:20 smithi016 ceph-mon[37215]: pgmap v1742: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1017 B/s rd, 4.8 KiB/s wr, 1 op/s 2024-03-26T04:51:21.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:20 smithi183 ceph-mon[33745]: pgmap v1742: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1017 B/s rd, 4.8 KiB/s wr, 1 op/s 2024-03-26T04:51:21.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:21 smithi016 ceph-mon[29101]: pool '9afc82fb-6d6b-4f10-901a-c3cc23ea025f' is full (reached quota's max_bytes: 100 B) 2024-03-26T04:51:21.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:21 smithi016 ceph-mon[29101]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T04:51:21.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:21 smithi016 ceph-mon[29101]: osdmap e806: 8 total, 8 up, 8 in 2024-03-26T04:51:21.909 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:21 smithi016 ceph-mon[37215]: pool '9afc82fb-6d6b-4f10-901a-c3cc23ea025f' is full (reached quota's max_bytes: 100 B) 2024-03-26T04:51:21.910 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:21 smithi016 ceph-mon[37215]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T04:51:21.910 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:21 smithi016 ceph-mon[37215]: osdmap e806: 8 total, 8 up, 8 in 2024-03-26T04:51:22.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:21 smithi183 ceph-mon[33745]: pool '9afc82fb-6d6b-4f10-901a-c3cc23ea025f' is full (reached quota's max_bytes: 100 B) 2024-03-26T04:51:22.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:21 smithi183 ceph-mon[33745]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T04:51:22.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:21 smithi183 ceph-mon[33745]: osdmap e806: 8 total, 8 up, 8 in 2024-03-26T04:51:22.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:22 smithi016 ceph-mon[29101]: pgmap v1744: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 977 B/s rd, 4.6 KiB/s wr, 1 op/s 2024-03-26T04:51:22.909 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:22 smithi016 ceph-mon[37215]: pgmap v1744: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 977 B/s rd, 4.6 KiB/s wr, 1 op/s 2024-03-26T04:51:23.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:22 smithi183 ceph-mon[33745]: pgmap v1744: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 977 B/s rd, 4.6 KiB/s wr, 1 op/s 2024-03-26T04:51:24.038 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:51:23 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:51:25.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:24 smithi183 ceph-mon[33745]: pgmap v1745: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 850 B/s rd, 0 op/s 2024-03-26T04:51:25.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:24 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:51:25.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:24 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:51:25.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:24 smithi016 ceph-mon[37215]: pgmap v1745: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 850 B/s rd, 0 op/s 2024-03-26T04:51:25.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:24 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:51:25.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:24 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:51:25.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:24 smithi016 ceph-mon[29101]: pgmap v1745: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 850 B/s rd, 0 op/s 2024-03-26T04:51:25.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:24 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:51:25.160 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:24 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:51:27.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:26 smithi183 ceph-mon[33745]: pgmap v1746: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:51:27.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:26 smithi016 ceph-mon[37215]: pgmap v1746: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:51:27.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:26 smithi016 ceph-mon[29101]: pgmap v1746: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-26T04:51:29.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:28 smithi183 ceph-mon[33745]: pgmap v1747: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s 2024-03-26T04:51:29.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:28 smithi016 ceph-mon[29101]: pgmap v1747: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s 2024-03-26T04:51:29.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:28 smithi016 ceph-mon[37215]: pgmap v1747: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1022 B/s rd, 0 op/s 2024-03-26T04:51:29.159 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:51:28 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:51:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:51:31.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:30 smithi183 ceph-mon[33745]: pgmap v1748: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-26T04:51:31.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:30 smithi016 ceph-mon[29101]: pgmap v1748: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-26T04:51:31.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:30 smithi016 ceph-mon[37215]: pgmap v1748: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-26T04:51:33.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:32 smithi183 ceph-mon[33745]: pgmap v1749: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-03-26T04:51:33.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:32 smithi016 ceph-mon[29101]: pgmap v1749: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-03-26T04:51:33.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:32 smithi016 ceph-mon[37215]: pgmap v1749: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 900 B/s rd, 0 op/s 2024-03-26T04:51:34.038 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:51:33 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:51:35.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:34 smithi183 ceph-mon[33745]: pgmap v1750: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:51:35.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:34 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:51:35.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:34 smithi016 ceph-mon[29101]: pgmap v1750: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:51:35.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:34 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:51:35.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:34 smithi016 ceph-mon[37215]: pgmap v1750: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:51:35.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:34 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:51:37.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:36 smithi183 ceph-mon[33745]: pgmap v1751: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T04:51:37.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:36 smithi016 ceph-mon[29101]: pgmap v1751: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T04:51:37.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:36 smithi016 ceph-mon[37215]: pgmap v1751: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 0 op/s 2024-03-26T04:51:39.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:38 smithi183 ceph-mon[33745]: pgmap v1752: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:51:39.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:38 smithi016 ceph-mon[29101]: pgmap v1752: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:51:39.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:38 smithi016 ceph-mon[37215]: pgmap v1752: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:51:39.159 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:51:38 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:51:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:51:40.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:39 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:51:40.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:39 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:51:40.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:39 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:51:41.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:40 smithi183 ceph-mon[33745]: pgmap v1753: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:51:41.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:40 smithi016 ceph-mon[29101]: pgmap v1753: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:51:41.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:40 smithi016 ceph-mon[37215]: pgmap v1753: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:51:43.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:42 smithi183 ceph-mon[33745]: pgmap v1754: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:51:43.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:42 smithi016 ceph-mon[37215]: pgmap v1754: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:51:43.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:42 smithi016 ceph-mon[29101]: pgmap v1754: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:51:44.038 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:51:43 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:51:45.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:44 smithi183 ceph-mon[33745]: pgmap v1755: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:51:45.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:44 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:51:45.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:44 smithi016 ceph-mon[29101]: pgmap v1755: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:51:45.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:44 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:51:45.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:44 smithi016 ceph-mon[37215]: pgmap v1755: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:51:45.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:44 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:51:47.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:46 smithi183 ceph-mon[33745]: pgmap v1756: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:51:47.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:46 smithi016 ceph-mon[29101]: pgmap v1756: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:51:47.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:46 smithi016 ceph-mon[37215]: pgmap v1756: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:51:48.654 INFO:tasks.workunit.client.0.smithi016.stderr:+ pid=151037 2024-03-26T04:51:48.654 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph osd pool set-quota 9afc82fb-6d6b-4f10-901a-c3cc23ea025f max_bytes 0 2024-03-26T04:51:48.654 INFO:tasks.workunit.client.0.smithi016.stderr:+ rados -p 9afc82fb-6d6b-4f10-901a-c3cc23ea025f put two /etc/passwd 2024-03-26T04:51:48.751 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:48.754+0000 7fa68615e640 1 -- 172.21.15.16:0/3912640308 >> v1:172.21.15.16:6790/0 conn(0x7fa680100560 legacy=0x7fa680100940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:51:48.751 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:48.754+0000 7fa68615e640 1 -- 172.21.15.16:0/3912640308 shutdown_connections 2024-03-26T04:51:48.751 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:48.754+0000 7fa68615e640 1 -- 172.21.15.16:0/3912640308 >> 172.21.15.16:0/3912640308 conn(0x7fa6800fbde0 msgr2=0x7fa6800fe200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:51:48.751 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:48.754+0000 7fa68615e640 1 -- 172.21.15.16:0/3912640308 shutdown_connections 2024-03-26T04:51:48.751 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:48.754+0000 7fa68615e640 1 -- 172.21.15.16:0/3912640308 wait complete. 2024-03-26T04:51:48.751 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:48.754+0000 7fa68615e640 1 Processor -- start 2024-03-26T04:51:48.754 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:48.754+0000 7fa68615e640 1 -- start start 2024-03-26T04:51:48.754 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:48.755+0000 7fa68615e640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa6801a6270 con 0x7fa680100560 2024-03-26T04:51:48.754 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:48.755+0000 7fa68615e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa6801a7470 con 0x7fa6801091e0 2024-03-26T04:51:48.754 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:48.755+0000 7fa68615e640 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa6801a8670 con 0x7fa68010cd50 2024-03-26T04:51:48.754 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:48.755+0000 7fa67f7fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa680100560 0x7fa680105860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59864/0 (socket says 172.21.15.16:59864) 2024-03-26T04:51:48.754 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:48.755+0000 7fa67f7fe640 1 -- 172.21.15.16:0/1105463316 learned_addr learned my addr 172.21.15.16:0/1105463316 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:51:48.754 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:48.755+0000 7fa67cff9640 1 -- 172.21.15.16:0/1105463316 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 9707099 0 0) 0x7fa6801a6270 con 0x7fa680100560 2024-03-26T04:51:48.754 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:48.755+0000 7fa67cff9640 1 -- 172.21.15.16:0/1105463316 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa668003520 con 0x7fa680100560 2024-03-26T04:51:48.754 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:48.755+0000 7fa67cff9640 1 -- 172.21.15.16:0/1105463316 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2809851150 0 0) 0x7fa6801a8670 con 0x7fa68010cd50 2024-03-26T04:51:48.754 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:48.755+0000 7fa67cff9640 1 -- 172.21.15.16:0/1105463316 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6801a6270 con 0x7fa68010cd50 2024-03-26T04:51:48.755 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:48.755+0000 7fa67cff9640 1 -- 172.21.15.16:0/1105463316 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 30492599 0 0) 0x7fa6801a7470 con 0x7fa6801091e0 2024-03-26T04:51:48.755 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:48.755+0000 7fa67cff9640 1 -- 172.21.15.16:0/1105463316 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6801a8670 con 0x7fa6801091e0 2024-03-26T04:51:48.755 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:48.755+0000 7fa67cff9640 1 -- 172.21.15.16:0/1105463316 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 724233520 0 0) 0x7fa6801a6270 con 0x7fa68010cd50 2024-03-26T04:51:48.755 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:48.755+0000 7fa67cff9640 1 -- 172.21.15.16:0/1105463316 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6801a7470 con 0x7fa68010cd50 2024-03-26T04:51:48.755 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:48.755+0000 7fa67cff9640 1 -- 172.21.15.16:0/1105463316 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fa6780019b0 con 0x7fa68010cd50 2024-03-26T04:51:48.755 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:48.755+0000 7fa67cff9640 1 -- 172.21.15.16:0/1105463316 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1911642084 0 0) 0x7fa668003520 con 0x7fa680100560 2024-03-26T04:51:48.755 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:48.755+0000 7fa67cff9640 1 -- 172.21.15.16:0/1105463316 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6801a6270 con 0x7fa680100560 2024-03-26T04:51:48.755 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:48.755+0000 7fa67cff9640 1 -- 172.21.15.16:0/1105463316 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fa670004320 con 0x7fa680100560 2024-03-26T04:51:48.755 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:48.755+0000 7fa67cff9640 1 -- 172.21.15.16:0/1105463316 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3119818844 0 0) 0x7fa6801a8670 con 0x7fa6801091e0 2024-03-26T04:51:48.755 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:48.755+0000 7fa67cff9640 1 -- 172.21.15.16:0/1105463316 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa668003520 con 0x7fa6801091e0 2024-03-26T04:51:48.755 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:48.755+0000 7fa67cff9640 1 -- 172.21.15.16:0/1105463316 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fa674001870 con 0x7fa6801091e0 2024-03-26T04:51:48.755 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:48.755+0000 7fa67cff9640 1 -- 172.21.15.16:0/1105463316 <== mon.2 v1:172.21.15.16:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2168565960 0 0) 0x7fa6801a7470 con 0x7fa68010cd50 2024-03-26T04:51:48.755 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:48.755+0000 7fa67cff9640 1 -- 172.21.15.16:0/1105463316 >> v1:172.21.15.183:6789/0 conn(0x7fa6801091e0 legacy=0x7fa6801a10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:51:48.755 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:48.755+0000 7fa67cff9640 1 -- 172.21.15.16:0/1105463316 >> v1:172.21.15.16:6789/0 conn(0x7fa680100560 legacy=0x7fa680105860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:51:48.756 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:48.755+0000 7fa67cff9640 1 -- 172.21.15.16:0/1105463316 --> v1:172.21.15.16:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6801a9870 con 0x7fa68010cd50 2024-03-26T04:51:48.756 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:48.756+0000 7fa68615e640 1 -- 172.21.15.16:0/1105463316 --> v1:172.21.15.16:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6801a6460 con 0x7fa68010cd50 2024-03-26T04:51:48.756 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:48.756+0000 7fa68615e640 1 -- 172.21.15.16:0/1105463316 --> v1:172.21.15.16:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6801a68f0 con 0x7fa68010cd50 2024-03-26T04:51:48.756 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:48.756+0000 7fa67cff9640 1 -- 172.21.15.16:0/1105463316 <== mon.2 v1:172.21.15.16:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7fa678003e00 con 0x7fa68010cd50 2024-03-26T04:51:48.756 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:48.756+0000 7fa67cff9640 1 -- 172.21.15.16:0/1105463316 <== mon.2 v1:172.21.15.16:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fa678004c50 con 0x7fa68010cd50 2024-03-26T04:51:48.756 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:48.757+0000 7fa68615e640 1 -- 172.21.15.16:0/1105463316 --> v1:172.21.15.16:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa66c005080 con 0x7fa68010cd50 2024-03-26T04:51:48.756 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:48.757+0000 7fa67cff9640 1 -- 172.21.15.16:0/1105463316 <== mon.2 v1:172.21.15.16:6790/0 7 ==== mgrmap(e 22) v1 ==== 91420+0+0 (unknown 3095259778 0 0) 0x7fa678003970 con 0x7fa68010cd50 2024-03-26T04:51:48.756 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:48.758+0000 7fa67cff9640 1 -- 172.21.15.16:0/1105463316 <== mon.2 v1:172.21.15.16:6790/0 8 ==== osd_map(806..806 src has 253..806) v4 ==== 7798+0+0 (unknown 1932404787 0 0) 0x7fa678087f50 con 0x7fa68010cd50 2024-03-26T04:51:48.757 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:48.758+0000 7fa67cff9640 1 -- 172.21.15.16:0/1105463316 --> v1:172.21.15.16:6790/0 -- mon_subscribe({osdmap=807}) v3 -- 0x7fa6801a7470 con 0x7fa68010cd50 2024-03-26T04:51:48.759 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:48.762+0000 7fa67cff9640 1 -- 172.21.15.16:0/1105463316 <== mon.2 v1:172.21.15.16:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fa678054bf0 con 0x7fa68010cd50 2024-03-26T04:51:48.945 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:48.948+0000 7fa68615e640 1 -- 172.21.15.16:0/1105463316 --> v1:172.21.15.16:6790/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "0"} v 0) v1 -- 0x7fa66c005330 con 0x7fa68010cd50 2024-03-26T04:51:49.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:48 smithi183 ceph-mon[33745]: pgmap v1757: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:51:49.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:48 smithi016 ceph-mon[37215]: pgmap v1757: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:51:49.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:48 smithi016 ceph-mon[29101]: pgmap v1757: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:51:49.159 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:51:48 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:51:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:51:49.750 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:49.753+0000 7fa67cff9640 1 -- 172.21.15.16:0/1105463316 <== mon.2 v1:172.21.15.16:6790/0 10 ==== osd_map(807..807 src has 253..807) v4 ==== 628+0+0 (unknown 2342120614 0 0) 0x7fa678055520 con 0x7fa68010cd50 2024-03-26T04:51:49.750 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:49.753+0000 7fa67cff9640 1 -- 172.21.15.16:0/1105463316 --> v1:172.21.15.16:6790/0 -- mon_subscribe({osdmap=808}) v3 -- 0x7fa6801a6270 con 0x7fa68010cd50 2024-03-26T04:51:49.753 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:49.757+0000 7fa67cff9640 1 -- 172.21.15.16:0/1105463316 <== mon.2 v1:172.21.15.16:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool 9afc82fb-6d6b-4f10-901a-c3cc23ea025f v807) v1 ==== 217+0+0 (unknown 3847581779 0 0) 0x7fa678052f60 con 0x7fa68010cd50 2024-03-26T04:51:49.869 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:49.872+0000 7fa68615e640 1 -- 172.21.15.16:0/1105463316 --> v1:172.21.15.16:6790/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "0"} v 0) v1 -- 0x7fa66c000c40 con 0x7fa68010cd50 2024-03-26T04:51:50.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:49 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1105463316' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T04:51:50.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:49 smithi183 ceph-mon[33745]: from='client.56954 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T04:51:50.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:49 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1105463316' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T04:51:50.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:49 smithi016 ceph-mon[37215]: from='client.56954 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T04:51:50.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:49 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1105463316' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T04:51:50.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:49 smithi016 ceph-mon[29101]: from='client.56954 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T04:51:50.751 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.754+0000 7fa67cff9640 1 -- 172.21.15.16:0/1105463316 <== mon.2 v1:172.21.15.16:6790/0 12 ==== osd_map(808..808 src has 253..808) v4 ==== 628+0+0 (unknown 2887352847 0 0) 0x7fa678058f70 con 0x7fa68010cd50 2024-03-26T04:51:50.752 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.755+0000 7fa67cff9640 1 -- 172.21.15.16:0/1105463316 --> v1:172.21.15.16:6790/0 -- mon_subscribe({osdmap=809}) v3 -- 0x7fa6801a8670 con 0x7fa68010cd50 2024-03-26T04:51:50.756 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.759+0000 7fa67cff9640 1 -- 172.21.15.16:0/1105463316 <== mon.2 v1:172.21.15.16:6790/0 13 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool 9afc82fb-6d6b-4f10-901a-c3cc23ea025f v808) v1 ==== 217+0+0 (unknown 1321067661 0 0) 0x7fa678054780 con 0x7fa68010cd50 2024-03-26T04:51:50.756 INFO:tasks.workunit.client.0.smithi016.stderr:set-quota max_bytes = 0 for pool 9afc82fb-6d6b-4f10-901a-c3cc23ea025f 2024-03-26T04:51:50.760 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.763+0000 7fa68615e640 1 -- 172.21.15.16:0/1105463316 >> v1:172.21.15.16:6800/3868004154 conn(0x7fa66806e920 legacy=0x7fa668070de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:51:50.760 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.763+0000 7fa68615e640 1 -- 172.21.15.16:0/1105463316 >> v1:172.21.15.16:6790/0 conn(0x7fa68010cd50 legacy=0x7fa6801a4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:51:50.761 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.763+0000 7fa68615e640 1 -- 172.21.15.16:0/1105463316 shutdown_connections 2024-03-26T04:51:50.761 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.764+0000 7fa68615e640 1 -- 172.21.15.16:0/1105463316 >> 172.21.15.16:0/1105463316 conn(0x7fa6800fbde0 msgr2=0x7fa680110270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:51:50.761 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.764+0000 7fa68615e640 1 -- 172.21.15.16:0/1105463316 shutdown_connections 2024-03-26T04:51:50.761 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.764+0000 7fa68615e640 1 -- 172.21.15.16:0/1105463316 wait complete. 2024-03-26T04:51:50.772 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph osd pool set-quota 9afc82fb-6d6b-4f10-901a-c3cc23ea025f max_objects 0 2024-03-26T04:51:50.863 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.863+0000 7fa2da7c9640 1 -- 172.21.15.16:0/1877080979 >> v1:172.21.15.16:6790/0 conn(0x7fa2d41091c0 legacy=0x7fa2d410b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:51:50.863 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.863+0000 7fa2da7c9640 1 -- 172.21.15.16:0/1877080979 shutdown_connections 2024-03-26T04:51:50.863 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.863+0000 7fa2da7c9640 1 -- 172.21.15.16:0/1877080979 >> 172.21.15.16:0/1877080979 conn(0x7fa2d4076fa0 msgr2=0x7fa2d40fe300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:51:50.863 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.863+0000 7fa2da7c9640 1 -- 172.21.15.16:0/1877080979 shutdown_connections 2024-03-26T04:51:50.863 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.863+0000 7fa2da7c9640 1 -- 172.21.15.16:0/1877080979 wait complete. 2024-03-26T04:51:50.863 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.864+0000 7fa2da7c9640 1 Processor -- start 2024-03-26T04:51:50.864 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.864+0000 7fa2da7c9640 1 -- start start 2024-03-26T04:51:50.864 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.864+0000 7fa2da7c9640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa2d41a6120 con 0x7fa2d41091c0 2024-03-26T04:51:50.864 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.864+0000 7fa2da7c9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa2d41a72e0 con 0x7fa2d4100540 2024-03-26T04:51:50.864 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.864+0000 7fa2da7c9640 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fa2d41a84e0 con 0x7fa2d410cd30 2024-03-26T04:51:50.864 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.864+0000 7fa2d8d66640 1 --1- >> v1:172.21.15.16:6790/0 conn(0x7fa2d410cd30 0x7fa2d41a4820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6790/0 says I am v1:172.21.15.16:56310/0 (socket says 172.21.15.16:56310) 2024-03-26T04:51:50.864 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.864+0000 7fa2d8d66640 1 -- 172.21.15.16:0/740858003 learned_addr learned my addr 172.21.15.16:0/740858003 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:51:50.864 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.864+0000 7fa2d37fe640 1 --1- 172.21.15.16:0/740858003 >> v1:172.21.15.16:6789/0 conn(0x7fa2d41091c0 0x7fa2d41a10b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59876/0 (socket says 172.21.15.16:59876) 2024-03-26T04:51:50.865 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.865+0000 7fa2d17fa640 1 -- 172.21.15.16:0/740858003 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1775722550 0 0) 0x7fa2d41a84e0 con 0x7fa2d410cd30 2024-03-26T04:51:50.865 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.865+0000 7fa2d17fa640 1 -- 172.21.15.16:0/740858003 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2bc003520 con 0x7fa2d410cd30 2024-03-26T04:51:50.865 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.865+0000 7fa2d17fa640 1 -- 172.21.15.16:0/740858003 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4144150705 0 0) 0x7fa2d41a6120 con 0x7fa2d41091c0 2024-03-26T04:51:50.865 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.865+0000 7fa2d17fa640 1 -- 172.21.15.16:0/740858003 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2d41a84e0 con 0x7fa2d41091c0 2024-03-26T04:51:50.865 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.865+0000 7fa2d17fa640 1 -- 172.21.15.16:0/740858003 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 641437037 0 0) 0x7fa2bc003520 con 0x7fa2d410cd30 2024-03-26T04:51:50.865 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.865+0000 7fa2d17fa640 1 -- 172.21.15.16:0/740858003 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2d41a6120 con 0x7fa2d410cd30 2024-03-26T04:51:50.865 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.865+0000 7fa2d17fa640 1 -- 172.21.15.16:0/740858003 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fa2cc001d50 con 0x7fa2d410cd30 2024-03-26T04:51:50.865 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.865+0000 7fa2d17fa640 1 -- 172.21.15.16:0/740858003 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4196977430 0 0) 0x7fa2d41a72e0 con 0x7fa2d4100540 2024-03-26T04:51:50.865 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.865+0000 7fa2d17fa640 1 -- 172.21.15.16:0/740858003 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2bc003520 con 0x7fa2d4100540 2024-03-26T04:51:50.865 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.865+0000 7fa2d17fa640 1 -- 172.21.15.16:0/740858003 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3446765153 0 0) 0x7fa2d41a84e0 con 0x7fa2d41091c0 2024-03-26T04:51:50.865 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.865+0000 7fa2d17fa640 1 -- 172.21.15.16:0/740858003 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2d41a72e0 con 0x7fa2d41091c0 2024-03-26T04:51:50.865 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.865+0000 7fa2d17fa640 1 -- 172.21.15.16:0/740858003 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fa2c80043d0 con 0x7fa2d41091c0 2024-03-26T04:51:50.865 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.865+0000 7fa2d17fa640 1 -- 172.21.15.16:0/740858003 <== mon.2 v1:172.21.15.16:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2595657701 0 0) 0x7fa2d41a6120 con 0x7fa2d410cd30 2024-03-26T04:51:50.866 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.865+0000 7fa2d17fa640 1 -- 172.21.15.16:0/740858003 >> v1:172.21.15.183:6789/0 conn(0x7fa2d4100540 legacy=0x7fa2d4068e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:51:50.866 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.865+0000 7fa2d17fa640 1 -- 172.21.15.16:0/740858003 >> v1:172.21.15.16:6789/0 conn(0x7fa2d41091c0 legacy=0x7fa2d41a10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:51:50.866 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.865+0000 7fa2d17fa640 1 -- 172.21.15.16:0/740858003 --> v1:172.21.15.16:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d41a96e0 con 0x7fa2d410cd30 2024-03-26T04:51:50.866 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.865+0000 7fa2da7c9640 1 -- 172.21.15.16:0/740858003 --> v1:172.21.15.16:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2d41a6310 con 0x7fa2d410cd30 2024-03-26T04:51:50.866 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.865+0000 7fa2da7c9640 1 -- 172.21.15.16:0/740858003 --> v1:172.21.15.16:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2d41a67a0 con 0x7fa2d410cd30 2024-03-26T04:51:50.866 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.865+0000 7fa2d17fa640 1 -- 172.21.15.16:0/740858003 <== mon.2 v1:172.21.15.16:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7fa2cc003e00 con 0x7fa2d410cd30 2024-03-26T04:51:50.866 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.865+0000 7fa2d17fa640 1 -- 172.21.15.16:0/740858003 <== mon.2 v1:172.21.15.16:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fa2cc005f00 con 0x7fa2d410cd30 2024-03-26T04:51:50.866 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.866+0000 7fa2da7c9640 1 -- 172.21.15.16:0/740858003 --> v1:172.21.15.16:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2c0005080 con 0x7fa2d410cd30 2024-03-26T04:51:50.866 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.866+0000 7fa2d17fa640 1 -- 172.21.15.16:0/740858003 <== mon.2 v1:172.21.15.16:6790/0 7 ==== mgrmap(e 22) v1 ==== 91420+0+0 (unknown 3095259778 0 0) 0x7fa2cc003970 con 0x7fa2d410cd30 2024-03-26T04:51:50.867 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.867+0000 7fa2d17fa640 1 -- 172.21.15.16:0/740858003 <== mon.2 v1:172.21.15.16:6790/0 8 ==== osd_map(808..808 src has 253..808) v4 ==== 7798+0+0 (unknown 2927393063 0 0) 0x7fa2cc089260 con 0x7fa2d410cd30 2024-03-26T04:51:50.867 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.867+0000 7fa2d17fa640 1 -- 172.21.15.16:0/740858003 --> v1:172.21.15.16:6790/0 -- mon_subscribe({osdmap=809}) v3 -- 0x7fa2d41a6120 con 0x7fa2d410cd30 2024-03-26T04:51:50.868 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:50.872+0000 7fa2d17fa640 1 -- 172.21.15.16:0/740858003 <== mon.2 v1:172.21.15.16:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fa2cc055f60 con 0x7fa2d410cd30 2024-03-26T04:51:51.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:50 smithi183 ceph-mon[33745]: pgmap v1758: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:51:51.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:50 smithi183 ceph-mon[33745]: from='client.56954 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T04:51:51.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:50 smithi183 ceph-mon[33745]: osdmap e807: 8 total, 8 up, 8 in 2024-03-26T04:51:51.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:50 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1105463316' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T04:51:51.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:50 smithi183 ceph-mon[33745]: from='client.56954 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T04:51:51.052 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:51.056+0000 7fa2da7c9640 1 -- 172.21.15.16:0/740858003 --> v1:172.21.15.16:6790/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "0"} v 0) v1 -- 0x7fa2c0005330 con 0x7fa2d410cd30 2024-03-26T04:51:51.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:50 smithi016 ceph-mon[29101]: pgmap v1758: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:51:51.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:50 smithi016 ceph-mon[29101]: from='client.56954 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T04:51:51.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:50 smithi016 ceph-mon[29101]: osdmap e807: 8 total, 8 up, 8 in 2024-03-26T04:51:51.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:50 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1105463316' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T04:51:51.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:50 smithi016 ceph-mon[29101]: from='client.56954 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T04:51:51.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:50 smithi016 ceph-mon[37215]: pgmap v1758: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:51:51.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:50 smithi016 ceph-mon[37215]: from='client.56954 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T04:51:51.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:50 smithi016 ceph-mon[37215]: osdmap e807: 8 total, 8 up, 8 in 2024-03-26T04:51:51.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:50 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1105463316' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T04:51:51.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:50 smithi016 ceph-mon[37215]: from='client.56954 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T04:51:51.419 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:51.422+0000 7fa2d17fa640 1 -- 172.21.15.16:0/740858003 <== mon.2 v1:172.21.15.16:6790/0 10 ==== osd_map(809..809 src has 253..809) v4 ==== 628+0+0 (unknown 2087227520 0 0) 0x7fa2cc056890 con 0x7fa2d410cd30 2024-03-26T04:51:51.419 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:51.422+0000 7fa2d17fa640 1 -- 172.21.15.16:0/740858003 --> v1:172.21.15.16:6790/0 -- mon_subscribe({osdmap=810}) v3 -- 0x7fa2d41a84e0 con 0x7fa2d410cd30 2024-03-26T04:51:51.422 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:51.426+0000 7fa2d17fa640 1 -- 172.21.15.16:0/740858003 <== mon.2 v1:172.21.15.16:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool 9afc82fb-6d6b-4f10-901a-c3cc23ea025f v809) v1 ==== 221+0+0 (unknown 1159373796 0 0) 0x7fa2cc0542d0 con 0x7fa2d410cd30 2024-03-26T04:51:51.537 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:51.541+0000 7fa2da7c9640 1 -- 172.21.15.16:0/740858003 --> v1:172.21.15.16:6790/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "0"} v 0) v1 -- 0x7fa2c0000c40 con 0x7fa2d410cd30 2024-03-26T04:51:52.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:51 smithi183 ceph-mon[33745]: from='client.56954 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T04:51:52.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:51 smithi183 ceph-mon[33745]: osdmap e808: 8 total, 8 up, 8 in 2024-03-26T04:51:52.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:51 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/740858003' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T04:51:52.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:51 smithi183 ceph-mon[33745]: from='client.56960 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T04:51:52.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:51 smithi183 ceph-mon[33745]: pool '9afc82fb-6d6b-4f10-901a-c3cc23ea025f' no longer out of quota; removing NO_QUOTA flag 2024-03-26T04:51:52.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:51 smithi183 ceph-mon[33745]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T04:51:52.039 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:51 smithi183 ceph-mon[33745]: from='client.56960 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "0"}]': finished 2024-03-26T04:51:52.039 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:51 smithi183 ceph-mon[33745]: osdmap e809: 8 total, 8 up, 8 in 2024-03-26T04:51:52.039 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:51 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/740858003' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T04:51:52.039 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:51 smithi183 ceph-mon[33745]: from='client.56960 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T04:51:52.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:51 smithi016 ceph-mon[29101]: from='client.56954 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T04:51:52.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:51 smithi016 ceph-mon[29101]: osdmap e808: 8 total, 8 up, 8 in 2024-03-26T04:51:52.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:51 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/740858003' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T04:51:52.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:51 smithi016 ceph-mon[29101]: from='client.56960 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T04:51:52.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:51 smithi016 ceph-mon[29101]: pool '9afc82fb-6d6b-4f10-901a-c3cc23ea025f' no longer out of quota; removing NO_QUOTA flag 2024-03-26T04:51:52.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:51 smithi016 ceph-mon[29101]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T04:51:52.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:51 smithi016 ceph-mon[29101]: from='client.56960 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "0"}]': finished 2024-03-26T04:51:52.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:51 smithi016 ceph-mon[29101]: osdmap e809: 8 total, 8 up, 8 in 2024-03-26T04:51:52.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:51 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/740858003' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T04:51:52.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:51 smithi016 ceph-mon[29101]: from='client.56960 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T04:51:52.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:51 smithi016 ceph-mon[37215]: from='client.56954 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T04:51:52.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:51 smithi016 ceph-mon[37215]: osdmap e808: 8 total, 8 up, 8 in 2024-03-26T04:51:52.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:51 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/740858003' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T04:51:52.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:51 smithi016 ceph-mon[37215]: from='client.56960 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T04:51:52.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:51 smithi016 ceph-mon[37215]: pool '9afc82fb-6d6b-4f10-901a-c3cc23ea025f' no longer out of quota; removing NO_QUOTA flag 2024-03-26T04:51:52.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:51 smithi016 ceph-mon[37215]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T04:51:52.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:51 smithi016 ceph-mon[37215]: from='client.56960 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "0"}]': finished 2024-03-26T04:51:52.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:51 smithi016 ceph-mon[37215]: osdmap e809: 8 total, 8 up, 8 in 2024-03-26T04:51:52.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:51 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/740858003' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T04:51:52.160 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:51 smithi016 ceph-mon[37215]: from='client.56960 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T04:51:52.424 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.427+0000 7fa2d17fa640 1 -- 172.21.15.16:0/740858003 <== mon.2 v1:172.21.15.16:6790/0 12 ==== osd_map(810..810 src has 253..810) v4 ==== 628+0+0 (unknown 555062266 0 0) 0x7fa2cc05a2e0 con 0x7fa2d410cd30 2024-03-26T04:51:52.429 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.431+0000 7fa2d17fa640 1 -- 172.21.15.16:0/740858003 <== mon.2 v1:172.21.15.16:6790/0 13 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool 9afc82fb-6d6b-4f10-901a-c3cc23ea025f v810) v1 ==== 221+0+0 (unknown 3221727030 0 0) 0x7fa2cc055af0 con 0x7fa2d410cd30 2024-03-26T04:51:52.429 INFO:tasks.workunit.client.0.smithi016.stderr:set-quota max_objects = 0 for pool 9afc82fb-6d6b-4f10-901a-c3cc23ea025f 2024-03-26T04:51:52.432 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.435+0000 7fa2da7c9640 1 -- 172.21.15.16:0/740858003 >> v1:172.21.15.16:6800/3868004154 conn(0x7fa2bc06eba0 legacy=0x7fa2bc071060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:51:52.432 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.435+0000 7fa2da7c9640 1 -- 172.21.15.16:0/740858003 >> v1:172.21.15.16:6790/0 conn(0x7fa2d410cd30 legacy=0x7fa2d41a4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:51:52.432 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.435+0000 7fa2da7c9640 1 -- 172.21.15.16:0/740858003 shutdown_connections 2024-03-26T04:51:52.432 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.435+0000 7fa2da7c9640 1 -- 172.21.15.16:0/740858003 >> 172.21.15.16:0/740858003 conn(0x7fa2d4076fa0 msgr2=0x7fa2d410f1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:51:52.432 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.436+0000 7fa2da7c9640 1 -- 172.21.15.16:0/740858003 shutdown_connections 2024-03-26T04:51:52.432 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.436+0000 7fa2da7c9640 1 -- 172.21.15.16:0/740858003 wait complete. 2024-03-26T04:51:52.443 INFO:tasks.workunit.client.0.smithi016.stderr:+ wait 151037 2024-03-26T04:51:52.443 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' 0 -ne 0 ']' 2024-03-26T04:51:52.443 INFO:tasks.workunit.client.0.smithi016.stderr:+ true 2024-03-26T04:51:52.443 INFO:tasks.workunit.client.0.smithi016.stderr:+ rados -p 9afc82fb-6d6b-4f10-901a-c3cc23ea025f put three /etc/passwd 2024-03-26T04:51:52.479 INFO:tasks.workunit.client.0.smithi016.stderr:++ uuidgen 2024-03-26T04:51:52.480 INFO:tasks.workunit.client.0.smithi016.stderr:+ pp=dcd349bf-3588-4b0c-9f5d-114ed7e44baf 2024-03-26T04:51:52.480 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph osd pool create dcd349bf-3588-4b0c-9f5d-114ed7e44baf 12 2024-03-26T04:51:52.567 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.568+0000 7fd0eade9640 1 -- 172.21.15.16:0/2433039882 >> v1:172.21.15.16:6789/0 conn(0x7fd0e410de20 legacy=0x7fd0e41102e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:51:52.567 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.568+0000 7fd0eade9640 1 -- 172.21.15.16:0/2433039882 shutdown_connections 2024-03-26T04:51:52.567 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.568+0000 7fd0eade9640 1 -- 172.21.15.16:0/2433039882 >> 172.21.15.16:0/2433039882 conn(0x7fd0e40fce70 msgr2=0x7fd0e40ff290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:51:52.568 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.568+0000 7fd0eade9640 1 -- 172.21.15.16:0/2433039882 shutdown_connections 2024-03-26T04:51:52.568 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.568+0000 7fd0eade9640 1 -- 172.21.15.16:0/2433039882 wait complete. 2024-03-26T04:51:52.568 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.569+0000 7fd0eade9640 1 Processor -- start 2024-03-26T04:51:52.568 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.569+0000 7fd0eade9640 1 -- start start 2024-03-26T04:51:52.568 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.569+0000 7fd0eade9640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd0e41a6260 con 0x7fd0e4101630 2024-03-26T04:51:52.568 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.569+0000 7fd0eade9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd0e41a7460 con 0x7fd0e410de20 2024-03-26T04:51:52.568 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.569+0000 7fd0eade9640 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fd0e41a8660 con 0x7fd0e410a2b0 2024-03-26T04:51:52.568 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.569+0000 7fd0e8b85640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd0e4101630 0x7fd0e4106870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59910/0 (socket says 172.21.15.16:59910) 2024-03-26T04:51:52.568 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.569+0000 7fd0e8b85640 1 -- 172.21.15.16:0/1948053386 learned_addr learned my addr 172.21.15.16:0/1948053386 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:51:52.568 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.569+0000 7fd0e3fff640 1 --1- >> v1:172.21.15.16:6790/0 conn(0x7fd0e410a2b0 0x7fd0e41a1190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6790/0 says I am v1:172.21.15.16:56348/0 (socket says 172.21.15.16:56348) 2024-03-26T04:51:52.568 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.569+0000 7fd0e1ffb640 1 -- 172.21.15.16:0/1948053386 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2633953313 0 0) 0x7fd0e41a6260 con 0x7fd0e4101630 2024-03-26T04:51:52.568 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.569+0000 7fd0e1ffb640 1 -- 172.21.15.16:0/1948053386 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0d4003520 con 0x7fd0e4101630 2024-03-26T04:51:52.568 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.569+0000 7fd0e1ffb640 1 -- 172.21.15.16:0/1948053386 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1399282007 0 0) 0x7fd0e41a8660 con 0x7fd0e410a2b0 2024-03-26T04:51:52.568 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.569+0000 7fd0e1ffb640 1 -- 172.21.15.16:0/1948053386 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0e41a6260 con 0x7fd0e410a2b0 2024-03-26T04:51:52.569 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.570+0000 7fd0e1ffb640 1 -- 172.21.15.16:0/1948053386 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3605411953 0 0) 0x7fd0e41a7460 con 0x7fd0e410de20 2024-03-26T04:51:52.569 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.570+0000 7fd0e1ffb640 1 -- 172.21.15.16:0/1948053386 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0e41a8660 con 0x7fd0e410de20 2024-03-26T04:51:52.569 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.570+0000 7fd0e1ffb640 1 -- 172.21.15.16:0/1948053386 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1875458132 0 0) 0x7fd0d4003520 con 0x7fd0e4101630 2024-03-26T04:51:52.569 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.570+0000 7fd0e1ffb640 1 -- 172.21.15.16:0/1948053386 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd0e41a7460 con 0x7fd0e4101630 2024-03-26T04:51:52.569 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.570+0000 7fd0e1ffb640 1 -- 172.21.15.16:0/1948053386 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fd0d8001930 con 0x7fd0e4101630 2024-03-26T04:51:52.569 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.570+0000 7fd0e1ffb640 1 -- 172.21.15.16:0/1948053386 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2709268532 0 0) 0x7fd0e41a6260 con 0x7fd0e410a2b0 2024-03-26T04:51:52.569 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.570+0000 7fd0e1ffb640 1 -- 172.21.15.16:0/1948053386 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd0d4003520 con 0x7fd0e410a2b0 2024-03-26T04:51:52.569 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.570+0000 7fd0e1ffb640 1 -- 172.21.15.16:0/1948053386 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fd0d0001630 con 0x7fd0e410a2b0 2024-03-26T04:51:52.569 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.570+0000 7fd0e1ffb640 1 -- 172.21.15.16:0/1948053386 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2589044257 0 0) 0x7fd0e41a7460 con 0x7fd0e4101630 2024-03-26T04:51:52.569 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.570+0000 7fd0e1ffb640 1 -- 172.21.15.16:0/1948053386 >> v1:172.21.15.16:6790/0 conn(0x7fd0e410a2b0 legacy=0x7fd0e41a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:51:52.569 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.570+0000 7fd0e1ffb640 1 -- 172.21.15.16:0/1948053386 >> v1:172.21.15.183:6789/0 conn(0x7fd0e410de20 legacy=0x7fd0e41a4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:51:52.570 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.570+0000 7fd0e1ffb640 1 -- 172.21.15.16:0/1948053386 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0e41a9860 con 0x7fd0e4101630 2024-03-26T04:51:52.570 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.570+0000 7fd0eade9640 1 -- 172.21.15.16:0/1948053386 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0e41a8850 con 0x7fd0e4101630 2024-03-26T04:51:52.570 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.570+0000 7fd0e1ffb640 1 -- 172.21.15.16:0/1948053386 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7fd0d8001cc0 con 0x7fd0e4101630 2024-03-26T04:51:52.570 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.570+0000 7fd0e1ffb640 1 -- 172.21.15.16:0/1948053386 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fd0d8005ce0 con 0x7fd0e4101630 2024-03-26T04:51:52.570 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.570+0000 7fd0eade9640 1 -- 172.21.15.16:0/1948053386 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0e41a8ce0 con 0x7fd0e4101630 2024-03-26T04:51:52.572 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.571+0000 7fd0eade9640 1 -- 172.21.15.16:0/1948053386 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0e4102b00 con 0x7fd0e4101630 2024-03-26T04:51:52.572 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.571+0000 7fd0e1ffb640 1 -- 172.21.15.16:0/1948053386 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 22) v1 ==== 91420+0+0 (unknown 3095259778 0 0) 0x7fd0d80038c0 con 0x7fd0e4101630 2024-03-26T04:51:52.572 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.572+0000 7fd0e1ffb640 1 -- 172.21.15.16:0/1948053386 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(810..810 src has 253..810) v4 ==== 7798+0+0 (unknown 2277139707 0 0) 0x7fd0d80890c0 con 0x7fd0e4101630 2024-03-26T04:51:52.573 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.576+0000 7fd0e1ffb640 1 -- 172.21.15.16:0/1948053386 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fd0d8055d60 con 0x7fd0e4101630 2024-03-26T04:51:52.756 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:52.759+0000 7fd0eade9640 1 -- 172.21.15.16:0/1948053386 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "pg_num": 12} v 0) v1 -- 0x7fd0e4068660 con 0x7fd0e4101630 2024-03-26T04:51:53.288 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:53 smithi183 ceph-mon[33745]: pgmap v1762: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:51:53.288 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:53 smithi183 ceph-mon[33745]: from='client.56960 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "0"}]': finished 2024-03-26T04:51:53.288 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:53 smithi183 ceph-mon[33745]: osdmap e810: 8 total, 8 up, 8 in 2024-03-26T04:51:53.288 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:53 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1948053386' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "pg_num": 12}]: dispatch 2024-03-26T04:51:53.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:53 smithi016 ceph-mon[29101]: pgmap v1762: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:51:53.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:53 smithi016 ceph-mon[29101]: from='client.56960 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "0"}]': finished 2024-03-26T04:51:53.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:53 smithi016 ceph-mon[29101]: osdmap e810: 8 total, 8 up, 8 in 2024-03-26T04:51:53.409 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:53 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1948053386' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "pg_num": 12}]: dispatch 2024-03-26T04:51:53.409 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:53 smithi016 ceph-mon[37215]: pgmap v1762: 176 pgs: 176 active+clean; 721 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:51:53.409 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:53 smithi016 ceph-mon[37215]: from='client.56960 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "0"}]': finished 2024-03-26T04:51:53.409 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:53 smithi016 ceph-mon[37215]: osdmap e810: 8 total, 8 up, 8 in 2024-03-26T04:51:53.409 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:53 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1948053386' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "pg_num": 12}]: dispatch 2024-03-26T04:51:53.425 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.428+0000 7fd0e1ffb640 1 -- 172.21.15.16:0/1948053386 <== mon.0 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "pg_num": 12}]=0 pool 'dcd349bf-3588-4b0c-9f5d-114ed7e44baf' created v811) v1 ==== 176+0+0 (unknown 1827927413 0 0) 0x7fd0d80540d0 con 0x7fd0e4101630 2024-03-26T04:51:53.543 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.546+0000 7fd0eade9640 1 -- 172.21.15.16:0/1948053386 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "pg_num": 12} v 0) v1 -- 0x7fd0e4104e40 con 0x7fd0e4101630 2024-03-26T04:51:53.543 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.547+0000 7fd0e1ffb640 1 -- 172.21.15.16:0/1948053386 <== mon.0 v1:172.21.15.16:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "pg_num": 12}]=0 pool 'dcd349bf-3588-4b0c-9f5d-114ed7e44baf' already exists v811) v1 ==== 183+0+0 (unknown 694876961 0 0) 0x7fd0d80558f0 con 0x7fd0e4101630 2024-03-26T04:51:53.544 INFO:tasks.workunit.client.0.smithi016.stderr:pool 'dcd349bf-3588-4b0c-9f5d-114ed7e44baf' already exists 2024-03-26T04:51:53.545 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.549+0000 7fd0eade9640 1 -- 172.21.15.16:0/1948053386 >> v1:172.21.15.16:6800/3868004154 conn(0x7fd0d406e9c0 legacy=0x7fd0d4070e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:51:53.545 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.549+0000 7fd0eade9640 1 -- 172.21.15.16:0/1948053386 >> v1:172.21.15.16:6789/0 conn(0x7fd0e4101630 legacy=0x7fd0e4106870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:51:53.545 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.549+0000 7fd0eade9640 1 -- 172.21.15.16:0/1948053386 shutdown_connections 2024-03-26T04:51:53.546 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.549+0000 7fd0eade9640 1 -- 172.21.15.16:0/1948053386 >> 172.21.15.16:0/1948053386 conn(0x7fd0e40fce70 msgr2=0x7fd0e4110ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:51:53.546 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.549+0000 7fd0eade9640 1 -- 172.21.15.16:0/1948053386 shutdown_connections 2024-03-26T04:51:53.546 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.549+0000 7fd0eade9640 1 -- 172.21.15.16:0/1948053386 wait complete. 2024-03-26T04:51:53.554 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph osd pool application enable dcd349bf-3588-4b0c-9f5d-114ed7e44baf rados 2024-03-26T04:51:53.640 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.641+0000 7f5c0d730640 1 -- 172.21.15.16:0/372268709 >> v1:172.21.15.16:6790/0 conn(0x7f5c0810df50 legacy=0x7f5c08110340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:51:53.640 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.641+0000 7f5c0d730640 1 -- 172.21.15.16:0/372268709 shutdown_connections 2024-03-26T04:51:53.640 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.641+0000 7f5c0d730640 1 -- 172.21.15.16:0/372268709 >> 172.21.15.16:0/372268709 conn(0x7f5c08076fa0 msgr2=0x7f5c080fe300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:51:53.640 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.641+0000 7f5c0d730640 1 -- 172.21.15.16:0/372268709 shutdown_connections 2024-03-26T04:51:53.640 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.641+0000 7f5c0d730640 1 -- 172.21.15.16:0/372268709 wait complete. 2024-03-26T04:51:53.641 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.641+0000 7f5c0d730640 1 Processor -- start 2024-03-26T04:51:53.641 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.642+0000 7f5c0d730640 1 -- start start 2024-03-26T04:51:53.641 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.642+0000 7f5c0d730640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5c081a62f0 con 0x7f5c08102640 2024-03-26T04:51:53.641 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.642+0000 7f5c0d730640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5c081a74f0 con 0x7f5c0810a440 2024-03-26T04:51:53.641 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.642+0000 7f5c0d730640 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5c081a86f0 con 0x7f5c0810df50 2024-03-26T04:51:53.641 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.642+0000 7f5c06ffd640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5c08102640 0x7f5c08109c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59924/0 (socket says 172.21.15.16:59924) 2024-03-26T04:51:53.641 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.642+0000 7f5c077fe640 1 --1- >> v1:172.21.15.16:6790/0 conn(0x7f5c0810df50 0x7f5c081a49f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6790/0 says I am v1:172.21.15.16:56366/0 (socket says 172.21.15.16:56366) 2024-03-26T04:51:53.641 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.642+0000 7f5c06ffd640 1 -- 172.21.15.16:0/1295770730 learned_addr learned my addr 172.21.15.16:0/1295770730 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:51:53.641 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.642+0000 7f5beffff640 1 -- 172.21.15.16:0/1295770730 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3277419992 0 0) 0x7f5c081a86f0 con 0x7f5c0810df50 2024-03-26T04:51:53.641 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.642+0000 7f5beffff640 1 -- 172.21.15.16:0/1295770730 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bf0003520 con 0x7f5c0810df50 2024-03-26T04:51:53.642 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.642+0000 7f5beffff640 1 -- 172.21.15.16:0/1295770730 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1281821287 0 0) 0x7f5c081a62f0 con 0x7f5c08102640 2024-03-26T04:51:53.642 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.642+0000 7f5beffff640 1 -- 172.21.15.16:0/1295770730 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c081a86f0 con 0x7f5c08102640 2024-03-26T04:51:53.642 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.642+0000 7f5beffff640 1 -- 172.21.15.16:0/1295770730 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 344030903 0 0) 0x7f5c081a74f0 con 0x7f5c0810a440 2024-03-26T04:51:53.642 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.642+0000 7f5beffff640 1 -- 172.21.15.16:0/1295770730 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c081a62f0 con 0x7f5c0810a440 2024-03-26T04:51:53.642 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.642+0000 7f5beffff640 1 -- 172.21.15.16:0/1295770730 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3752497624 0 0) 0x7f5bf0003520 con 0x7f5c0810df50 2024-03-26T04:51:53.642 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.642+0000 7f5beffff640 1 -- 172.21.15.16:0/1295770730 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c081a74f0 con 0x7f5c0810df50 2024-03-26T04:51:53.642 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.642+0000 7f5beffff640 1 -- 172.21.15.16:0/1295770730 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f5bf8004660 con 0x7f5c0810df50 2024-03-26T04:51:53.642 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.642+0000 7f5beffff640 1 -- 172.21.15.16:0/1295770730 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 359927616 0 0) 0x7f5c081a86f0 con 0x7f5c08102640 2024-03-26T04:51:53.642 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.642+0000 7f5beffff640 1 -- 172.21.15.16:0/1295770730 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5bf0003520 con 0x7f5c08102640 2024-03-26T04:51:53.642 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.642+0000 7f5beffff640 1 -- 172.21.15.16:0/1295770730 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f5c000017f0 con 0x7f5c08102640 2024-03-26T04:51:53.642 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.642+0000 7f5beffff640 1 -- 172.21.15.16:0/1295770730 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2274400131 0 0) 0x7f5c081a62f0 con 0x7f5c0810a440 2024-03-26T04:51:53.642 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.642+0000 7f5beffff640 1 -- 172.21.15.16:0/1295770730 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c081a86f0 con 0x7f5c0810a440 2024-03-26T04:51:53.642 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.642+0000 7f5beffff640 1 -- 172.21.15.16:0/1295770730 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f5bfc001970 con 0x7f5c0810a440 2024-03-26T04:51:53.642 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.643+0000 7f5beffff640 1 -- 172.21.15.16:0/1295770730 <== mon.2 v1:172.21.15.16:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 535924689 0 0) 0x7f5c081a74f0 con 0x7f5c0810df50 2024-03-26T04:51:53.643 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.643+0000 7f5beffff640 1 -- 172.21.15.16:0/1295770730 >> v1:172.21.15.183:6789/0 conn(0x7f5c0810a440 legacy=0x7f5c081a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:51:53.643 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.643+0000 7f5beffff640 1 -- 172.21.15.16:0/1295770730 >> v1:172.21.15.16:6789/0 conn(0x7f5c08102640 legacy=0x7f5c08109c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:51:53.643 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.643+0000 7f5beffff640 1 -- 172.21.15.16:0/1295770730 --> v1:172.21.15.16:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c081a98f0 con 0x7f5c0810df50 2024-03-26T04:51:53.643 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.643+0000 7f5c0d730640 1 -- 172.21.15.16:0/1295770730 --> v1:172.21.15.16:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c081a88e0 con 0x7f5c0810df50 2024-03-26T04:51:53.643 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.643+0000 7f5c0d730640 1 -- 172.21.15.16:0/1295770730 --> v1:172.21.15.16:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c081a8d70 con 0x7f5c0810df50 2024-03-26T04:51:53.643 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.643+0000 7f5beffff640 1 -- 172.21.15.16:0/1295770730 <== mon.2 v1:172.21.15.16:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f5bf8001e70 con 0x7f5c0810df50 2024-03-26T04:51:53.643 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.643+0000 7f5beffff640 1 -- 172.21.15.16:0/1295770730 <== mon.2 v1:172.21.15.16:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f5bf8005040 con 0x7f5c0810df50 2024-03-26T04:51:53.643 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.644+0000 7f5c0d730640 1 -- 172.21.15.16:0/1295770730 --> v1:172.21.15.16:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bf4005080 con 0x7f5c0810df50 2024-03-26T04:51:53.644 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.644+0000 7f5beffff640 1 -- 172.21.15.16:0/1295770730 <== mon.2 v1:172.21.15.16:6790/0 7 ==== mgrmap(e 22) v1 ==== 91420+0+0 (unknown 3095259778 0 0) 0x7f5bf80051a0 con 0x7f5c0810df50 2024-03-26T04:51:53.644 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.645+0000 7f5beffff640 1 -- 172.21.15.16:0/1295770730 <== mon.2 v1:172.21.15.16:6790/0 8 ==== osd_map(811..811 src has 253..811) v4 ==== 8173+0+0 (unknown 198445882 0 0) 0x7f5bf80894b0 con 0x7f5c0810df50 2024-03-26T04:51:53.645 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.649+0000 7f5beffff640 1 -- 172.21.15.16:0/1295770730 <== mon.2 v1:172.21.15.16:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f5bf8055d50 con 0x7f5c0810df50 2024-03-26T04:51:53.828 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:53.831+0000 7f5c0d730640 1 -- 172.21.15.16:0/1295770730 --> v1:172.21.15.16:6790/0 -- mon_command({"prefix": "osd pool application enable", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "app": "rados"} v 0) v1 -- 0x7f5bf4005330 con 0x7f5c0810df50 2024-03-26T04:51:54.038 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:51:53 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:51:54.436 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:54.439+0000 7f5beffff640 1 -- 172.21.15.16:0/1295770730 <== mon.2 v1:172.21.15.16:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "app": "rados"}]=0 enabled application 'rados' on pool 'dcd349bf-3588-4b0c-9f5d-114ed7e44baf' v812) v1 ==== 213+0+0 (unknown 2234496316 0 0) 0x7f5bf8054350 con 0x7f5c0810df50 2024-03-26T04:51:54.554 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:54.557+0000 7f5c0d730640 1 -- 172.21.15.16:0/1295770730 --> v1:172.21.15.16:6790/0 -- mon_command({"prefix": "osd pool application enable", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "app": "rados"} v 0) v1 -- 0x7f5bf4001090 con 0x7f5c0810df50 2024-03-26T04:51:54.788 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:54 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1948053386' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "pg_num": 12}]': finished 2024-03-26T04:51:54.788 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:54 smithi183 ceph-mon[33745]: osdmap e811: 8 total, 8 up, 8 in 2024-03-26T04:51:54.788 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:54 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1948053386' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "pg_num": 12}]: dispatch 2024-03-26T04:51:54.788 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:54 smithi183 ceph-mon[33745]: pgmap v1765: 188 pgs: 12 unknown, 176 active+clean; 732 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 3.1 KiB/s wr, 1 op/s 2024-03-26T04:51:54.788 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:54 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:51:54.788 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:54 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1295770730' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "app": "rados"}]: dispatch 2024-03-26T04:51:54.788 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:54 smithi183 ceph-mon[33745]: from='client.56981 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "app": "rados"}]: dispatch 2024-03-26T04:51:54.788 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:54 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:51:54.789 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:54 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:51:54.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:54 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1948053386' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "pg_num": 12}]': finished 2024-03-26T04:51:54.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:54 smithi016 ceph-mon[29101]: osdmap e811: 8 total, 8 up, 8 in 2024-03-26T04:51:54.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:54 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1948053386' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "pg_num": 12}]: dispatch 2024-03-26T04:51:54.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:54 smithi016 ceph-mon[29101]: pgmap v1765: 188 pgs: 12 unknown, 176 active+clean; 732 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 3.1 KiB/s wr, 1 op/s 2024-03-26T04:51:54.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:54 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:51:54.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:54 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1295770730' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "app": "rados"}]: dispatch 2024-03-26T04:51:54.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:54 smithi016 ceph-mon[29101]: from='client.56981 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "app": "rados"}]: dispatch 2024-03-26T04:51:54.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:54 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:51:54.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:54 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:51:54.910 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:54 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1948053386' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "pg_num": 12}]': finished 2024-03-26T04:51:54.910 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:54 smithi016 ceph-mon[37215]: osdmap e811: 8 total, 8 up, 8 in 2024-03-26T04:51:54.910 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:54 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1948053386' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "pg_num": 12}]: dispatch 2024-03-26T04:51:54.910 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:54 smithi016 ceph-mon[37215]: pgmap v1765: 188 pgs: 12 unknown, 176 active+clean; 732 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 3.1 KiB/s wr, 1 op/s 2024-03-26T04:51:54.910 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:54 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:51:54.910 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:54 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1295770730' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "app": "rados"}]: dispatch 2024-03-26T04:51:54.910 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:54 smithi016 ceph-mon[37215]: from='client.56981 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "app": "rados"}]: dispatch 2024-03-26T04:51:54.910 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:54 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:51:54.910 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:54 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:51:55.440 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.443+0000 7f5beffff640 1 -- 172.21.15.16:0/1295770730 <== mon.2 v1:172.21.15.16:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "app": "rados"}]=0 enabled application 'rados' on pool 'dcd349bf-3588-4b0c-9f5d-114ed7e44baf' v813) v1 ==== 213+0+0 (unknown 3205383130 0 0) 0x7f5bf8055b70 con 0x7f5c0810df50 2024-03-26T04:51:55.441 INFO:tasks.workunit.client.0.smithi016.stderr:enabled application 'rados' on pool 'dcd349bf-3588-4b0c-9f5d-114ed7e44baf' 2024-03-26T04:51:55.443 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.447+0000 7f5c0d730640 1 -- 172.21.15.16:0/1295770730 >> v1:172.21.15.16:6800/3868004154 conn(0x7f5bf006ea80 legacy=0x7f5bf0070f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:51:55.443 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.447+0000 7f5c0d730640 1 -- 172.21.15.16:0/1295770730 >> v1:172.21.15.16:6790/0 conn(0x7f5c0810df50 legacy=0x7f5c081a49f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:51:55.444 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.447+0000 7f5c0d730640 1 -- 172.21.15.16:0/1295770730 shutdown_connections 2024-03-26T04:51:55.444 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.447+0000 7f5c0d730640 1 -- 172.21.15.16:0/1295770730 >> 172.21.15.16:0/1295770730 conn(0x7f5c08076fa0 msgr2=0x7f5c0810fa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:51:55.444 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.447+0000 7f5c0d730640 1 -- 172.21.15.16:0/1295770730 shutdown_connections 2024-03-26T04:51:55.444 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.447+0000 7f5c0d730640 1 -- 172.21.15.16:0/1295770730 wait complete. 2024-03-26T04:51:55.455 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph osd pool set-quota dcd349bf-3588-4b0c-9f5d-114ed7e44baf max_objects 10 2024-03-26T04:51:55.545 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.547+0000 7ff90c793640 1 -- 172.21.15.16:0/3533758758 >> v1:172.21.15.16:6789/0 conn(0x7ff90410a270 legacy=0x7ff90410c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:51:55.545 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.547+0000 7ff90c793640 1 -- 172.21.15.16:0/3533758758 shutdown_connections 2024-03-26T04:51:55.545 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.547+0000 7ff90c793640 1 -- 172.21.15.16:0/3533758758 >> 172.21.15.16:0/3533758758 conn(0x7ff9040755c0 msgr2=0x7ff9040759d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:51:55.545 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.547+0000 7ff90c793640 1 -- 172.21.15.16:0/3533758758 shutdown_connections 2024-03-26T04:51:55.545 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.547+0000 7ff90c793640 1 -- 172.21.15.16:0/3533758758 wait complete. 2024-03-26T04:51:55.545 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.548+0000 7ff90c793640 1 Processor -- start 2024-03-26T04:51:55.545 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.548+0000 7ff90c793640 1 -- start start 2024-03-26T04:51:55.545 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.548+0000 7ff90c793640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff9041a62c0 con 0x7ff90410a270 2024-03-26T04:51:55.545 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.548+0000 7ff90c793640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff9041a7480 con 0x7ff90410de40 2024-03-26T04:51:55.545 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.548+0000 7ff90c793640 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7ff9041a8680 con 0x7ff904107460 2024-03-26T04:51:55.545 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.548+0000 7ff90a52f640 1 --1- >> v1:172.21.15.16:6790/0 conn(0x7ff904107460 0x7ff904100430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6790/0 says I am v1:172.21.15.16:56384/0 (socket says 172.21.15.16:56384) 2024-03-26T04:51:55.546 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.548+0000 7ff909d2e640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff90410a270 0x7ff9041a1190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59952/0 (socket says 172.21.15.16:59952) 2024-03-26T04:51:55.546 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.548+0000 7ff90a52f640 1 -- 172.21.15.16:0/734562312 learned_addr learned my addr 172.21.15.16:0/734562312 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:51:55.548 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.549+0000 7ff8fb7fe640 1 -- 172.21.15.16:0/734562312 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3621125551 0 0) 0x7ff9041a62c0 con 0x7ff90410a270 2024-03-26T04:51:55.548 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.549+0000 7ff8fb7fe640 1 -- 172.21.15.16:0/734562312 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8f0004950 con 0x7ff90410a270 2024-03-26T04:51:55.548 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.549+0000 7ff8fb7fe640 1 -- 172.21.15.16:0/734562312 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2034759898 0 0) 0x7ff9041a8680 con 0x7ff904107460 2024-03-26T04:51:55.548 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.549+0000 7ff8fb7fe640 1 -- 172.21.15.16:0/734562312 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9041a62c0 con 0x7ff904107460 2024-03-26T04:51:55.548 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.549+0000 7ff8fb7fe640 1 -- 172.21.15.16:0/734562312 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2630869903 0 0) 0x7ff9041a7480 con 0x7ff90410de40 2024-03-26T04:51:55.548 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.549+0000 7ff8fb7fe640 1 -- 172.21.15.16:0/734562312 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9041a8680 con 0x7ff90410de40 2024-03-26T04:51:55.549 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.549+0000 7ff8fb7fe640 1 -- 172.21.15.16:0/734562312 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3010546291 0 0) 0x7ff8f0004950 con 0x7ff90410a270 2024-03-26T04:51:55.549 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.549+0000 7ff8fb7fe640 1 -- 172.21.15.16:0/734562312 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9041a7480 con 0x7ff90410a270 2024-03-26T04:51:55.549 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.549+0000 7ff8fb7fe640 1 -- 172.21.15.16:0/734562312 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7ff8f40014f0 con 0x7ff90410a270 2024-03-26T04:51:55.549 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.549+0000 7ff8fb7fe640 1 -- 172.21.15.16:0/734562312 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1228620085 0 0) 0x7ff9041a62c0 con 0x7ff904107460 2024-03-26T04:51:55.549 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.549+0000 7ff8fb7fe640 1 -- 172.21.15.16:0/734562312 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8f0004950 con 0x7ff904107460 2024-03-26T04:51:55.549 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.549+0000 7ff8fb7fe640 1 -- 172.21.15.16:0/734562312 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7ff900001850 con 0x7ff904107460 2024-03-26T04:51:55.549 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.549+0000 7ff8fb7fe640 1 -- 172.21.15.16:0/734562312 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 337964483 0 0) 0x7ff9041a8680 con 0x7ff90410de40 2024-03-26T04:51:55.549 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.549+0000 7ff8fb7fe640 1 -- 172.21.15.16:0/734562312 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9041a62c0 con 0x7ff90410de40 2024-03-26T04:51:55.549 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.549+0000 7ff8fb7fe640 1 -- 172.21.15.16:0/734562312 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7ff8fc001c30 con 0x7ff90410de40 2024-03-26T04:51:55.549 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.549+0000 7ff8fb7fe640 1 -- 172.21.15.16:0/734562312 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2558451601 0 0) 0x7ff9041a7480 con 0x7ff90410a270 2024-03-26T04:51:55.549 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.549+0000 7ff8fb7fe640 1 -- 172.21.15.16:0/734562312 >> v1:172.21.15.16:6790/0 conn(0x7ff904107460 legacy=0x7ff904100430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:51:55.549 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.549+0000 7ff8fb7fe640 1 -- 172.21.15.16:0/734562312 >> v1:172.21.15.183:6789/0 conn(0x7ff90410de40 legacy=0x7ff9041a4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:51:55.549 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.549+0000 7ff8fb7fe640 1 -- 172.21.15.16:0/734562312 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9041a9880 con 0x7ff90410a270 2024-03-26T04:51:55.549 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.549+0000 7ff90c793640 1 -- 172.21.15.16:0/734562312 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9041a8870 con 0x7ff90410a270 2024-03-26T04:51:55.550 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.549+0000 7ff90c793640 1 -- 172.21.15.16:0/734562312 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9041a8d00 con 0x7ff90410a270 2024-03-26T04:51:55.550 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.549+0000 7ff8fb7fe640 1 -- 172.21.15.16:0/734562312 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7ff8f4001ad0 con 0x7ff90410a270 2024-03-26T04:51:55.550 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.549+0000 7ff8fb7fe640 1 -- 172.21.15.16:0/734562312 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7ff8f4004d90 con 0x7ff90410a270 2024-03-26T04:51:55.550 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.550+0000 7ff90c793640 1 -- 172.21.15.16:0/734562312 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8e8005080 con 0x7ff90410a270 2024-03-26T04:51:55.550 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.551+0000 7ff8fb7fe640 1 -- 172.21.15.16:0/734562312 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 22) v1 ==== 91420+0+0 (unknown 3095259778 0 0) 0x7ff8f4004ef0 con 0x7ff90410a270 2024-03-26T04:51:55.550 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.551+0000 7ff8fb7fe640 1 -- 172.21.15.16:0/734562312 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(813..813 src has 253..813) v4 ==== 8186+0+0 (unknown 172007915 0 0) 0x7ff8f40891a0 con 0x7ff90410a270 2024-03-26T04:51:55.552 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.556+0000 7ff8fb7fe640 1 -- 172.21.15.16:0/734562312 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7ff8f4055d20 con 0x7ff90410a270 2024-03-26T04:51:55.738 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:55.742+0000 7ff90c793640 1 -- 172.21.15.16:0/734562312 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "field": "max_objects", "val": "10"} v 0) v1 -- 0x7ff8e8005330 con 0x7ff90410a270 2024-03-26T04:51:55.788 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:55 smithi183 ceph-mon[33745]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:51:55.788 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:55 smithi183 ceph-mon[33745]: from='client.56981 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "app": "rados"}]': finished 2024-03-26T04:51:55.788 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:55 smithi183 ceph-mon[33745]: osdmap e812: 8 total, 8 up, 8 in 2024-03-26T04:51:55.788 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:55 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/1295770730' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "app": "rados"}]: dispatch 2024-03-26T04:51:55.788 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:55 smithi183 ceph-mon[33745]: from='client.56981 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "app": "rados"}]: dispatch 2024-03-26T04:51:55.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:55 smithi016 ceph-mon[29101]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:51:55.908 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:55 smithi016 ceph-mon[29101]: from='client.56981 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "app": "rados"}]': finished 2024-03-26T04:51:55.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:55 smithi016 ceph-mon[29101]: osdmap e812: 8 total, 8 up, 8 in 2024-03-26T04:51:55.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:55 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/1295770730' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "app": "rados"}]: dispatch 2024-03-26T04:51:55.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:55 smithi016 ceph-mon[29101]: from='client.56981 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "app": "rados"}]: dispatch 2024-03-26T04:51:55.909 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:55 smithi016 ceph-mon[37215]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:51:55.909 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:55 smithi016 ceph-mon[37215]: from='client.56981 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "app": "rados"}]': finished 2024-03-26T04:51:55.909 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:55 smithi016 ceph-mon[37215]: osdmap e812: 8 total, 8 up, 8 in 2024-03-26T04:51:55.909 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:55 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/1295770730' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "app": "rados"}]: dispatch 2024-03-26T04:51:55.909 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:55 smithi016 ceph-mon[37215]: from='client.56981 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "app": "rados"}]: dispatch 2024-03-26T04:51:56.441 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:56.444+0000 7ff8fb7fe640 1 -- 172.21.15.16:0/734562312 <== mon.0 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool dcd349bf-3588-4b0c-9f5d-114ed7e44baf v814) v1 ==== 223+0+0 (unknown 3713602835 0 0) 0x7ff8f4054090 con 0x7ff90410a270 2024-03-26T04:51:56.556 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:56.559+0000 7ff90c793640 1 -- 172.21.15.16:0/734562312 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "field": "max_objects", "val": "10"} v 0) v1 -- 0x7ff8e8001090 con 0x7ff90410a270 2024-03-26T04:51:56.788 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:56 smithi183 ceph-mon[33745]: from='client.56981 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "app": "rados"}]': finished 2024-03-26T04:51:56.788 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:56 smithi183 ceph-mon[33745]: osdmap e813: 8 total, 8 up, 8 in 2024-03-26T04:51:56.788 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:56 smithi183 ceph-mon[33745]: pgmap v1768: 188 pgs: 12 unknown, 176 active+clean; 732 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-26T04:51:56.788 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:56 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/734562312' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T04:51:56.908 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:56 smithi016 ceph-mon[37215]: from='client.56981 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "app": "rados"}]': finished 2024-03-26T04:51:56.909 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:56 smithi016 ceph-mon[37215]: osdmap e813: 8 total, 8 up, 8 in 2024-03-26T04:51:56.909 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:56 smithi016 ceph-mon[37215]: pgmap v1768: 188 pgs: 12 unknown, 176 active+clean; 732 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-26T04:51:56.909 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:56 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/734562312' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T04:51:56.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:56 smithi016 ceph-mon[29101]: from='client.56981 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "app": "rados"}]': finished 2024-03-26T04:51:56.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:56 smithi016 ceph-mon[29101]: osdmap e813: 8 total, 8 up, 8 in 2024-03-26T04:51:56.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:56 smithi016 ceph-mon[29101]: pgmap v1768: 188 pgs: 12 unknown, 176 active+clean; 732 KiB data, 876 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-26T04:51:56.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:56 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/734562312' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T04:51:57.447 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:57.450+0000 7ff8fb7fe640 1 -- 172.21.15.16:0/734562312 <== mon.0 v1:172.21.15.16:6789/0 11 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "field": "max_objects", "val": "10"}]=0 set-quota max_objects = 10 for pool dcd349bf-3588-4b0c-9f5d-114ed7e44baf v815) v1 ==== 223+0+0 (unknown 2484908131 0 0) 0x7ff8f40558b0 con 0x7ff90410a270 2024-03-26T04:51:57.447 INFO:tasks.workunit.client.0.smithi016.stderr:set-quota max_objects = 10 for pool dcd349bf-3588-4b0c-9f5d-114ed7e44baf 2024-03-26T04:51:57.450 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:57.454+0000 7ff90c793640 1 -- 172.21.15.16:0/734562312 >> v1:172.21.15.16:6800/3868004154 conn(0x7ff8f006e9e0 legacy=0x7ff8f0070ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:51:57.450 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:57.454+0000 7ff90c793640 1 -- 172.21.15.16:0/734562312 >> v1:172.21.15.16:6789/0 conn(0x7ff90410a270 legacy=0x7ff9041a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:51:57.451 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:57.454+0000 7ff90c793640 1 -- 172.21.15.16:0/734562312 shutdown_connections 2024-03-26T04:51:57.451 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:57.454+0000 7ff90c793640 1 -- 172.21.15.16:0/734562312 >> 172.21.15.16:0/734562312 conn(0x7ff9040755c0 msgr2=0x7ff904108ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:51:57.451 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:57.454+0000 7ff90c793640 1 -- 172.21.15.16:0/734562312 shutdown_connections 2024-03-26T04:51:57.451 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:51:57.454+0000 7ff90c793640 1 -- 172.21.15.16:0/734562312 wait complete. 2024-03-26T04:51:57.463 INFO:tasks.workunit.client.0.smithi016.stderr:+ sleep 30 2024-03-26T04:51:57.788 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:57 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/734562312' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "field": "max_objects", "val": "10"}]': finished 2024-03-26T04:51:57.788 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:57 smithi183 ceph-mon[33745]: osdmap e814: 8 total, 8 up, 8 in 2024-03-26T04:51:57.788 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:57 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/734562312' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T04:51:57.908 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:57 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/734562312' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "field": "max_objects", "val": "10"}]': finished 2024-03-26T04:51:57.909 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:57 smithi016 ceph-mon[37215]: osdmap e814: 8 total, 8 up, 8 in 2024-03-26T04:51:57.909 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:57 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/734562312' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T04:51:57.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:57 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/734562312' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "field": "max_objects", "val": "10"}]': finished 2024-03-26T04:51:57.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:57 smithi016 ceph-mon[29101]: osdmap e814: 8 total, 8 up, 8 in 2024-03-26T04:51:57.909 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:57 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/734562312' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "field": "max_objects", "val": "10"}]: dispatch 2024-03-26T04:51:58.742 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/734562312' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "field": "max_objects", "val": "10"}]': finished 2024-03-26T04:51:58.742 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:58 smithi016 ceph-mon[37215]: osdmap e815: 8 total, 8 up, 8 in 2024-03-26T04:51:58.742 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:51:58 smithi016 ceph-mon[37215]: pgmap v1771: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-26T04:51:58.742 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/734562312' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "field": "max_objects", "val": "10"}]': finished 2024-03-26T04:51:58.742 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:58 smithi016 ceph-mon[29101]: osdmap e815: 8 total, 8 up, 8 in 2024-03-26T04:51:58.742 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:51:58 smithi016 ceph-mon[29101]: pgmap v1771: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-26T04:51:58.788 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/734562312' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "field": "max_objects", "val": "10"}]': finished 2024-03-26T04:51:58.788 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:58 smithi183 ceph-mon[33745]: osdmap e815: 8 total, 8 up, 8 in 2024-03-26T04:51:58.788 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:51:58 smithi183 ceph-mon[33745]: pgmap v1771: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 3.0 KiB/s wr, 1 op/s 2024-03-26T04:51:59.158 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:51:58 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:51:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:52:00.977 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:00 smithi016 ceph-mon[37215]: pgmap v1772: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 978 B/s rd, 2.3 KiB/s wr, 1 op/s 2024-03-26T04:52:00.977 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:00 smithi016 ceph-mon[29101]: pgmap v1772: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 978 B/s rd, 2.3 KiB/s wr, 1 op/s 2024-03-26T04:52:01.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:00 smithi183 ceph-mon[33745]: pgmap v1772: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 978 B/s rd, 2.3 KiB/s wr, 1 op/s 2024-03-26T04:52:02.037 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:01 smithi183 ceph-mon[33745]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:52:02.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:01 smithi016 ceph-mon[37215]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:52:02.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:01 smithi016 ceph-mon[29101]: Health check update: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-26T04:52:03.037 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:02 smithi183 ceph-mon[33745]: pgmap v1773: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 822 B/s rd, 1.9 KiB/s wr, 0 op/s 2024-03-26T04:52:03.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:02 smithi016 ceph-mon[29101]: pgmap v1773: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 822 B/s rd, 1.9 KiB/s wr, 0 op/s 2024-03-26T04:52:03.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:02 smithi016 ceph-mon[37215]: pgmap v1773: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 822 B/s rd, 1.9 KiB/s wr, 0 op/s 2024-03-26T04:52:04.038 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:52:03 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:52:05.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:04 smithi183 ceph-mon[33745]: pgmap v1774: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 1 op/s 2024-03-26T04:52:05.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:04 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:52:05.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:04 smithi016 ceph-mon[37215]: pgmap v1774: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 1 op/s 2024-03-26T04:52:05.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:04 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:52:05.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:04 smithi016 ceph-mon[29101]: pgmap v1774: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 1 op/s 2024-03-26T04:52:05.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:04 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:52:07.037 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:06 smithi183 ceph-mon[33745]: pgmap v1775: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 1 op/s 2024-03-26T04:52:07.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:06 smithi016 ceph-mon[37215]: pgmap v1775: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 1 op/s 2024-03-26T04:52:07.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:06 smithi016 ceph-mon[29101]: pgmap v1775: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 1 op/s 2024-03-26T04:52:09.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:08 smithi183 ceph-mon[33745]: pgmap v1776: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1002 B/s rd, 0 op/s 2024-03-26T04:52:09.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:08 smithi016 ceph-mon[37215]: pgmap v1776: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1002 B/s rd, 0 op/s 2024-03-26T04:52:09.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:08 smithi016 ceph-mon[29101]: pgmap v1776: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1002 B/s rd, 0 op/s 2024-03-26T04:52:09.159 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:52:08 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:52:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:52:10.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:09 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:52:10.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:09 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:52:10.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:09 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:52:11.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:10 smithi183 ceph-mon[33745]: pgmap v1777: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:52:11.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:10 smithi016 ceph-mon[29101]: pgmap v1777: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:52:11.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:10 smithi016 ceph-mon[37215]: pgmap v1777: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:52:13.037 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:12 smithi183 ceph-mon[33745]: pgmap v1778: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:52:13.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:12 smithi016 ceph-mon[37215]: pgmap v1778: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:52:13.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:12 smithi016 ceph-mon[29101]: pgmap v1778: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:52:14.037 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:13 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:52:14.038 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:52:13 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:52:14.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:13 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:52:14.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:13 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:52:15.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:14 smithi183 ceph-mon[33745]: pgmap v1779: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:15.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:14 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:52:15.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:14 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:52:15.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:14 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:52:15.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:14 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:52:15.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:14 smithi016 ceph-mon[29101]: pgmap v1779: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:15.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:14 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:52:15.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:14 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:52:15.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:14 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:52:15.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:14 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:52:15.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:14 smithi016 ceph-mon[37215]: pgmap v1779: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:15.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:14 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:52:15.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:14 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:52:15.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:14 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:52:15.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:14 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:52:17.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:16 smithi183 ceph-mon[33745]: pgmap v1780: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:52:17.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:16 smithi016 ceph-mon[37215]: pgmap v1780: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:52:17.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:16 smithi016 ceph-mon[29101]: pgmap v1780: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:52:19.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:18 smithi183 ceph-mon[33745]: pgmap v1781: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:19.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:18 smithi016 ceph-mon[37215]: pgmap v1781: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:19.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:18 smithi016 ceph-mon[29101]: pgmap v1781: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:19.159 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:52:18 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:52:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:52:21.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:20 smithi183 ceph-mon[33745]: pgmap v1782: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:52:21.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:20 smithi016 ceph-mon[37215]: pgmap v1782: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:52:21.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:20 smithi016 ceph-mon[29101]: pgmap v1782: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:52:23.037 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:22 smithi183 ceph-mon[33745]: pgmap v1783: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:52:23.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:22 smithi016 ceph-mon[37215]: pgmap v1783: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:52:23.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:22 smithi016 ceph-mon[29101]: pgmap v1783: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:52:24.038 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:52:23 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:52:25.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:24 smithi183 ceph-mon[33745]: pgmap v1784: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:25.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:24 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:52:25.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:24 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:52:25.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:24 smithi016 ceph-mon[37215]: pgmap v1784: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:25.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:24 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:52:25.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:24 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:52:25.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:24 smithi016 ceph-mon[29101]: pgmap v1784: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:25.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:24 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:52:25.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:24 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:52:27.037 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:26 smithi183 ceph-mon[33745]: pgmap v1785: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:52:27.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:26 smithi016 ceph-mon[37215]: pgmap v1785: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:52:27.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:26 smithi016 ceph-mon[29101]: pgmap v1785: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:52:27.464 INFO:tasks.workunit.client.0.smithi016.stderr:++ seq 1 10 2024-03-26T04:52:27.465 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in `seq 1 10` 2024-03-26T04:52:27.465 INFO:tasks.workunit.client.0.smithi016.stderr:+ rados -p dcd349bf-3588-4b0c-9f5d-114ed7e44baf put obj1 /etc/passwd 2024-03-26T04:52:27.505 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in `seq 1 10` 2024-03-26T04:52:27.505 INFO:tasks.workunit.client.0.smithi016.stderr:+ rados -p dcd349bf-3588-4b0c-9f5d-114ed7e44baf put obj2 /etc/passwd 2024-03-26T04:52:27.540 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in `seq 1 10` 2024-03-26T04:52:27.541 INFO:tasks.workunit.client.0.smithi016.stderr:+ rados -p dcd349bf-3588-4b0c-9f5d-114ed7e44baf put obj3 /etc/passwd 2024-03-26T04:52:27.575 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in `seq 1 10` 2024-03-26T04:52:27.575 INFO:tasks.workunit.client.0.smithi016.stderr:+ rados -p dcd349bf-3588-4b0c-9f5d-114ed7e44baf put obj4 /etc/passwd 2024-03-26T04:52:27.609 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in `seq 1 10` 2024-03-26T04:52:27.609 INFO:tasks.workunit.client.0.smithi016.stderr:+ rados -p dcd349bf-3588-4b0c-9f5d-114ed7e44baf put obj5 /etc/passwd 2024-03-26T04:52:27.642 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in `seq 1 10` 2024-03-26T04:52:27.642 INFO:tasks.workunit.client.0.smithi016.stderr:+ rados -p dcd349bf-3588-4b0c-9f5d-114ed7e44baf put obj6 /etc/passwd 2024-03-26T04:52:27.678 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in `seq 1 10` 2024-03-26T04:52:27.678 INFO:tasks.workunit.client.0.smithi016.stderr:+ rados -p dcd349bf-3588-4b0c-9f5d-114ed7e44baf put obj7 /etc/passwd 2024-03-26T04:52:27.707 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in `seq 1 10` 2024-03-26T04:52:27.707 INFO:tasks.workunit.client.0.smithi016.stderr:+ rados -p dcd349bf-3588-4b0c-9f5d-114ed7e44baf put obj8 /etc/passwd 2024-03-26T04:52:27.739 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in `seq 1 10` 2024-03-26T04:52:27.739 INFO:tasks.workunit.client.0.smithi016.stderr:+ rados -p dcd349bf-3588-4b0c-9f5d-114ed7e44baf put obj9 /etc/passwd 2024-03-26T04:52:27.774 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in `seq 1 10` 2024-03-26T04:52:27.774 INFO:tasks.workunit.client.0.smithi016.stderr:+ rados -p dcd349bf-3588-4b0c-9f5d-114ed7e44baf put obj10 /etc/passwd 2024-03-26T04:52:27.806 INFO:tasks.workunit.client.0.smithi016.stderr:+ sleep 30 2024-03-26T04:52:29.037 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:28 smithi183 ceph-mon[33745]: pgmap v1786: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:29.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:28 smithi016 ceph-mon[37215]: pgmap v1786: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:29.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:28 smithi016 ceph-mon[29101]: pgmap v1786: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:29.159 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:52:28 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:52:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:52:31.037 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:30 smithi183 ceph-mon[33745]: pgmap v1787: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:52:31.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:30 smithi016 ceph-mon[29101]: pgmap v1787: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:52:31.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:30 smithi016 ceph-mon[37215]: pgmap v1787: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:52:33.037 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:32 smithi183 ceph-mon[33745]: pgmap v1788: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:52:33.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:32 smithi016 ceph-mon[37215]: pgmap v1788: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:52:33.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:32 smithi016 ceph-mon[29101]: pgmap v1788: 188 pgs: 188 active+clean; 744 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:52:34.038 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:52:33 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:52:35.037 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:34 smithi183 ceph-mon[33745]: pgmap v1789: 188 pgs: 188 active+clean; 859 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-03-26T04:52:35.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:34 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:52:35.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:34 smithi016 ceph-mon[37215]: pgmap v1789: 188 pgs: 188 active+clean; 859 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-03-26T04:52:35.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:34 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:52:35.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:34 smithi016 ceph-mon[29101]: pgmap v1789: 188 pgs: 188 active+clean; 859 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 10 KiB/s wr, 2 op/s 2024-03-26T04:52:35.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:34 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:52:37.037 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:36 smithi183 ceph-mon[33745]: pgmap v1790: 188 pgs: 188 active+clean; 859 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 10 KiB/s wr, 1 op/s 2024-03-26T04:52:37.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:36 smithi183 ceph-mon[33745]: pool 'dcd349bf-3588-4b0c-9f5d-114ed7e44baf' is full (reached quota's max_objects: 10) 2024-03-26T04:52:37.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:36 smithi183 ceph-mon[33745]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T04:52:37.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:36 smithi183 ceph-mon[33745]: osdmap e816: 8 total, 8 up, 8 in 2024-03-26T04:52:37.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:36 smithi016 ceph-mon[37215]: pgmap v1790: 188 pgs: 188 active+clean; 859 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 10 KiB/s wr, 1 op/s 2024-03-26T04:52:37.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:36 smithi016 ceph-mon[37215]: pool 'dcd349bf-3588-4b0c-9f5d-114ed7e44baf' is full (reached quota's max_objects: 10) 2024-03-26T04:52:37.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:36 smithi016 ceph-mon[37215]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T04:52:37.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:36 smithi016 ceph-mon[37215]: osdmap e816: 8 total, 8 up, 8 in 2024-03-26T04:52:37.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:36 smithi016 ceph-mon[29101]: pgmap v1790: 188 pgs: 188 active+clean; 859 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 10 KiB/s wr, 1 op/s 2024-03-26T04:52:37.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:36 smithi016 ceph-mon[29101]: pool 'dcd349bf-3588-4b0c-9f5d-114ed7e44baf' is full (reached quota's max_objects: 10) 2024-03-26T04:52:37.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:36 smithi016 ceph-mon[29101]: Health check failed: 1 pool(s) full (POOL_FULL) 2024-03-26T04:52:37.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:36 smithi016 ceph-mon[29101]: osdmap e816: 8 total, 8 up, 8 in 2024-03-26T04:52:39.037 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:38 smithi183 ceph-mon[33745]: pgmap v1792: 188 pgs: 188 active+clean; 859 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T04:52:39.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:38 smithi016 ceph-mon[37215]: pgmap v1792: 188 pgs: 188 active+clean; 859 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T04:52:39.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:38 smithi016 ceph-mon[29101]: pgmap v1792: 188 pgs: 188 active+clean; 859 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T04:52:39.159 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:52:38 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:52:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:52:40.287 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:40 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:52:40.288 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:40 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:52:40.288 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:40 smithi183 ceph-mon[33745]: pgmap v1793: 188 pgs: 188 active+clean; 859 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T04:52:40.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:40 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:52:40.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:40 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:52:40.408 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:40 smithi016 ceph-mon[29101]: pgmap v1793: 188 pgs: 188 active+clean; 859 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T04:52:40.409 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:40 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:52:40.409 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:40 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:52:40.409 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:40 smithi016 ceph-mon[37215]: pgmap v1793: 188 pgs: 188 active+clean; 859 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T04:52:43.037 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:42 smithi183 ceph-mon[33745]: pgmap v1794: 188 pgs: 188 active+clean; 859 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T04:52:43.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:42 smithi016 ceph-mon[37215]: pgmap v1794: 188 pgs: 188 active+clean; 859 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T04:52:43.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:42 smithi016 ceph-mon[29101]: pgmap v1794: 188 pgs: 188 active+clean; 859 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 12 KiB/s wr, 1 op/s 2024-03-26T04:52:44.287 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:52:43 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:52:45.037 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:44 smithi183 ceph-mon[33745]: pgmap v1795: 188 pgs: 188 active+clean; 859 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:52:45.037 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:44 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:52:45.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:44 smithi016 ceph-mon[29101]: pgmap v1795: 188 pgs: 188 active+clean; 859 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:52:45.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:44 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:52:45.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:44 smithi016 ceph-mon[37215]: pgmap v1795: 188 pgs: 188 active+clean; 859 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:52:45.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:44 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:52:47.037 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:46 smithi183 ceph-mon[33745]: pgmap v1796: 188 pgs: 188 active+clean; 859 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:52:47.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:46 smithi016 ceph-mon[29101]: pgmap v1796: 188 pgs: 188 active+clean; 859 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:52:47.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:46 smithi016 ceph-mon[37215]: pgmap v1796: 188 pgs: 188 active+clean; 859 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:52:49.037 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:48 smithi183 ceph-mon[33745]: pgmap v1797: 188 pgs: 188 active+clean; 859 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T04:52:49.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:48 smithi016 ceph-mon[29101]: pgmap v1797: 188 pgs: 188 active+clean; 859 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T04:52:49.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:48 smithi016 ceph-mon[37215]: pgmap v1797: 188 pgs: 188 active+clean; 859 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-26T04:52:49.159 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:52:48 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:52:48] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:52:51.037 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:50 smithi183 ceph-mon[33745]: pgmap v1798: 188 pgs: 188 active+clean; 859 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:52:51.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:50 smithi016 ceph-mon[29101]: pgmap v1798: 188 pgs: 188 active+clean; 859 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:52:51.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:50 smithi016 ceph-mon[37215]: pgmap v1798: 188 pgs: 188 active+clean; 859 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:52:53.037 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:52 smithi183 ceph-mon[33745]: pgmap v1799: 188 pgs: 188 active+clean; 859 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:52:53.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:52 smithi016 ceph-mon[29101]: pgmap v1799: 188 pgs: 188 active+clean; 859 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:52:53.164 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:52 smithi016 ceph-mon[37215]: pgmap v1799: 188 pgs: 188 active+clean; 859 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:52:54.287 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:52:53 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:52:55.037 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:54 smithi183 ceph-mon[33745]: pgmap v1800: 188 pgs: 188 active+clean; 859 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:55.037 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:54 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:52:55.037 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:54 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:52:55.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:54 smithi016 ceph-mon[29101]: pgmap v1800: 188 pgs: 188 active+clean; 859 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:55.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:54 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:52:55.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:54 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:52:55.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:54 smithi016 ceph-mon[37215]: pgmap v1800: 188 pgs: 188 active+clean; 859 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:55.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:54 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:52:55.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:54 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:52:57.037 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:56 smithi183 ceph-mon[33745]: pgmap v1801: 188 pgs: 188 active+clean; 859 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:52:57.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:56 smithi016 ceph-mon[29101]: pgmap v1801: 188 pgs: 188 active+clean; 859 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:52:57.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:56 smithi016 ceph-mon[37215]: pgmap v1801: 188 pgs: 188 active+clean; 859 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:52:57.807 INFO:tasks.workunit.client.0.smithi016.stderr:+ rados -p 9afc82fb-6d6b-4f10-901a-c3cc23ea025f put threemore /etc/passwd 2024-03-26T04:52:57.845 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph osd pool set-quota 9afc82fb-6d6b-4f10-901a-c3cc23ea025f max_bytes 0 2024-03-26T04:52:57.937 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:57.938+0000 7f38aa1a5640 1 -- 172.21.15.16:0/4158482249 >> v1:172.21.15.16:6789/0 conn(0x7f38a41025f0 legacy=0x7f38a41029d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:52:57.937 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:57.938+0000 7f38aa1a5640 1 -- 172.21.15.16:0/4158482249 shutdown_connections 2024-03-26T04:52:57.937 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:57.938+0000 7f38aa1a5640 1 -- 172.21.15.16:0/4158482249 >> 172.21.15.16:0/4158482249 conn(0x7f38a40fbd90 msgr2=0x7f38a40fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:52:57.937 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:57.938+0000 7f38aa1a5640 1 -- 172.21.15.16:0/4158482249 shutdown_connections 2024-03-26T04:52:57.937 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:57.938+0000 7f38aa1a5640 1 -- 172.21.15.16:0/4158482249 wait complete. 2024-03-26T04:52:57.938 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:57.939+0000 7f38aa1a5640 1 Processor -- start 2024-03-26T04:52:57.938 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:57.939+0000 7f38aa1a5640 1 -- start start 2024-03-26T04:52:57.938 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:57.939+0000 7f38aa1a5640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f38a41a6150 con 0x7f38a41025f0 2024-03-26T04:52:57.938 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:57.939+0000 7f38aa1a5640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f38a41a7350 con 0x7f38a410a3f0 2024-03-26T04:52:57.938 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:57.939+0000 7f38aa1a5640 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f38a41a8550 con 0x7f38a410df00 2024-03-26T04:52:57.938 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:57.939+0000 7f38a3fff640 1 --1- >> v1:172.21.15.16:6790/0 conn(0x7f38a410df00 0x7f38a41a48d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6790/0 says I am v1:172.21.15.16:56074/0 (socket says 172.21.15.16:56074) 2024-03-26T04:52:57.938 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:57.939+0000 7f38a37fe640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f38a41025f0 0x7f38a4109a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:32986/0 (socket says 172.21.15.16:32986) 2024-03-26T04:52:57.938 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:57.939+0000 7f38a3fff640 1 -- 172.21.15.16:0/614852167 learned_addr learned my addr 172.21.15.16:0/614852167 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:52:57.938 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:57.940+0000 7f38a0ff9640 1 -- 172.21.15.16:0/614852167 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2686490569 0 0) 0x7f38a41a6150 con 0x7f38a41025f0 2024-03-26T04:52:57.938 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:57.940+0000 7f38a0ff9640 1 -- 172.21.15.16:0/614852167 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f388c003520 con 0x7f38a41025f0 2024-03-26T04:52:57.938 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:57.940+0000 7f38a0ff9640 1 -- 172.21.15.16:0/614852167 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1836131828 0 0) 0x7f38a41a8550 con 0x7f38a410df00 2024-03-26T04:52:57.938 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:57.940+0000 7f38a0ff9640 1 -- 172.21.15.16:0/614852167 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38a41a6150 con 0x7f38a410df00 2024-03-26T04:52:57.939 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:57.940+0000 7f38a0ff9640 1 -- 172.21.15.16:0/614852167 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 295013944 0 0) 0x7f38a41a7350 con 0x7f38a410a3f0 2024-03-26T04:52:57.939 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:57.940+0000 7f38a0ff9640 1 -- 172.21.15.16:0/614852167 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38a41a8550 con 0x7f38a410a3f0 2024-03-26T04:52:57.939 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:57.940+0000 7f38a0ff9640 1 -- 172.21.15.16:0/614852167 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3964733573 0 0) 0x7f388c003520 con 0x7f38a41025f0 2024-03-26T04:52:57.939 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:57.940+0000 7f38a0ff9640 1 -- 172.21.15.16:0/614852167 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38a41a7350 con 0x7f38a41025f0 2024-03-26T04:52:57.939 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:57.940+0000 7f38a0ff9640 1 -- 172.21.15.16:0/614852167 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f38940044b0 con 0x7f38a41025f0 2024-03-26T04:52:57.939 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:57.940+0000 7f38a0ff9640 1 -- 172.21.15.16:0/614852167 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2691107763 0 0) 0x7f38a41a6150 con 0x7f38a410df00 2024-03-26T04:52:57.939 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:57.940+0000 7f38a0ff9640 1 -- 172.21.15.16:0/614852167 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f388c003520 con 0x7f38a410df00 2024-03-26T04:52:57.939 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:57.940+0000 7f38a0ff9640 1 -- 172.21.15.16:0/614852167 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f389c001c40 con 0x7f38a410df00 2024-03-26T04:52:57.939 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:57.940+0000 7f38a0ff9640 1 -- 172.21.15.16:0/614852167 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1866224507 0 0) 0x7f38a41a8550 con 0x7f38a410a3f0 2024-03-26T04:52:57.939 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:57.940+0000 7f38a0ff9640 1 -- 172.21.15.16:0/614852167 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38a41a6150 con 0x7f38a410a3f0 2024-03-26T04:52:57.939 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:57.940+0000 7f38a0ff9640 1 -- 172.21.15.16:0/614852167 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f38980019f0 con 0x7f38a410a3f0 2024-03-26T04:52:57.939 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:57.940+0000 7f38a0ff9640 1 -- 172.21.15.16:0/614852167 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4182703611 0 0) 0x7f38a41a7350 con 0x7f38a41025f0 2024-03-26T04:52:57.940 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:57.940+0000 7f38a0ff9640 1 -- 172.21.15.16:0/614852167 >> v1:172.21.15.16:6790/0 conn(0x7f38a410df00 legacy=0x7f38a41a48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:52:57.940 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:57.940+0000 7f38a0ff9640 1 -- 172.21.15.16:0/614852167 >> v1:172.21.15.183:6789/0 conn(0x7f38a410a3f0 legacy=0x7f38a41a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:52:57.940 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:57.940+0000 7f38a0ff9640 1 -- 172.21.15.16:0/614852167 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a41a9750 con 0x7f38a41025f0 2024-03-26T04:52:57.940 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:57.940+0000 7f38aa1a5640 1 -- 172.21.15.16:0/614852167 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38a41a6340 con 0x7f38a41025f0 2024-03-26T04:52:57.940 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:57.940+0000 7f38aa1a5640 1 -- 172.21.15.16:0/614852167 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f38a41a67d0 con 0x7f38a41025f0 2024-03-26T04:52:57.940 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:57.940+0000 7f38a0ff9640 1 -- 172.21.15.16:0/614852167 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f38940011a0 con 0x7f38a41025f0 2024-03-26T04:52:57.940 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:57.940+0000 7f38a0ff9640 1 -- 172.21.15.16:0/614852167 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f3894004c40 con 0x7f38a41025f0 2024-03-26T04:52:57.940 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:57.941+0000 7f38aa1a5640 1 -- 172.21.15.16:0/614852167 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3890005080 con 0x7f38a41025f0 2024-03-26T04:52:57.941 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:57.942+0000 7f38a0ff9640 1 -- 172.21.15.16:0/614852167 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 22) v1 ==== 91420+0+0 (unknown 3095259778 0 0) 0x7f3894004e80 con 0x7f38a41025f0 2024-03-26T04:52:57.941 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:57.942+0000 7f38a0ff9640 1 -- 172.21.15.16:0/614852167 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(816..816 src has 253..816) v4 ==== 8186+0+0 (unknown 1952170303 0 0) 0x7f38940890d0 con 0x7f38a41025f0 2024-03-26T04:52:57.941 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:57.942+0000 7f38a0ff9640 1 -- 172.21.15.16:0/614852167 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=817}) v3 -- 0x7f38a41a7350 con 0x7f38a41025f0 2024-03-26T04:52:57.943 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:57.947+0000 7f38a0ff9640 1 -- 172.21.15.16:0/614852167 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f3894055d30 con 0x7f38a41025f0 2024-03-26T04:52:58.127 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:58.131+0000 7f38aa1a5640 1 -- 172.21.15.16:0/614852167 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "0"} v 0) v1 -- 0x7f3890005330 con 0x7f38a41025f0 2024-03-26T04:52:58.754 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:58.758+0000 7f38a0ff9640 1 -- 172.21.15.16:0/614852167 <== mon.0 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool 9afc82fb-6d6b-4f10-901a-c3cc23ea025f v817) v1 ==== 217+0+0 (unknown 567899976 0 0) 0x7f38940540a0 con 0x7f38a41025f0 2024-03-26T04:52:58.754 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:58.758+0000 7f38a0ff9640 1 -- 172.21.15.16:0/614852167 <== mon.0 v1:172.21.15.16:6789/0 11 ==== osd_map(817..817 src has 253..817) v4 ==== 628+0+0 (unknown 337828477 0 0) 0x7f3894001a70 con 0x7f38a41025f0 2024-03-26T04:52:58.755 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:58.758+0000 7f38a0ff9640 1 -- 172.21.15.16:0/614852167 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=818}) v3 -- 0x7f38a41a8550 con 0x7f38a41025f0 2024-03-26T04:52:58.871 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:58.875+0000 7f38aa1a5640 1 -- 172.21.15.16:0/614852167 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "0"} v 0) v1 -- 0x7f3890000c40 con 0x7f38a41025f0 2024-03-26T04:52:59.037 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:58 smithi183 ceph-mon[33745]: pgmap v1802: 188 pgs: 188 active+clean; 859 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:59.037 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:58 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/614852167' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T04:52:59.158 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:52:58 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:52:58] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:52:59.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:58 smithi016 ceph-mon[29101]: pgmap v1802: 188 pgs: 188 active+clean; 859 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:59.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:58 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/614852167' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T04:52:59.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:58 smithi016 ceph-mon[37215]: pgmap v1802: 188 pgs: 188 active+clean; 859 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:52:59.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:58 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/614852167' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T04:52:59.772 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.776+0000 7f38a0ff9640 1 -- 172.21.15.16:0/614852167 <== mon.0 v1:172.21.15.16:6789/0 12 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "0"}]=0 set-quota max_bytes = 0 for pool 9afc82fb-6d6b-4f10-901a-c3cc23ea025f v818) v1 ==== 217+0+0 (unknown 1473779691 0 0) 0x7f38940558c0 con 0x7f38a41025f0 2024-03-26T04:52:59.772 INFO:tasks.workunit.client.0.smithi016.stderr:set-quota max_bytes = 0 for pool 9afc82fb-6d6b-4f10-901a-c3cc23ea025f 2024-03-26T04:52:59.772 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.776+0000 7f38a0ff9640 1 -- 172.21.15.16:0/614852167 <== mon.0 v1:172.21.15.16:6789/0 13 ==== osd_map(818..818 src has 253..818) v4 ==== 628+0+0 (unknown 212285020 0 0) 0x7f389405a0b0 con 0x7f38a41025f0 2024-03-26T04:52:59.772 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.776+0000 7f38a0ff9640 1 -- 172.21.15.16:0/614852167 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=819}) v3 -- 0x7f388c07c3c0 con 0x7f38a41025f0 2024-03-26T04:52:59.776 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.780+0000 7f38aa1a5640 1 -- 172.21.15.16:0/614852167 >> v1:172.21.15.16:6800/3868004154 conn(0x7f388c06ea30 legacy=0x7f388c070ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:52:59.776 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.780+0000 7f38aa1a5640 1 -- 172.21.15.16:0/614852167 >> v1:172.21.15.16:6789/0 conn(0x7f38a41025f0 legacy=0x7f38a4109a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:52:59.776 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.780+0000 7f38aa1a5640 1 -- 172.21.15.16:0/614852167 shutdown_connections 2024-03-26T04:52:59.776 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.780+0000 7f38aa1a5640 1 -- 172.21.15.16:0/614852167 >> 172.21.15.16:0/614852167 conn(0x7f38a40fbd90 msgr2=0x7f38a40fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:52:59.777 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.780+0000 7f38aa1a5640 1 -- 172.21.15.16:0/614852167 shutdown_connections 2024-03-26T04:52:59.777 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.780+0000 7f38aa1a5640 1 -- 172.21.15.16:0/614852167 wait complete. 2024-03-26T04:52:59.792 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph osd pool set-quota 9afc82fb-6d6b-4f10-901a-c3cc23ea025f max_objects 0 2024-03-26T04:52:59.883 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.883+0000 7fc2b6f4f640 1 -- 172.21.15.16:0/2482638787 >> v1:172.21.15.16:6790/0 conn(0x7fc2b0100530 legacy=0x7fc2b0100910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:52:59.883 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.883+0000 7fc2b6f4f640 1 -- 172.21.15.16:0/2482638787 shutdown_connections 2024-03-26T04:52:59.883 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.883+0000 7fc2b6f4f640 1 -- 172.21.15.16:0/2482638787 >> 172.21.15.16:0/2482638787 conn(0x7fc2b00fbd90 msgr2=0x7fc2b00fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:52:59.883 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.883+0000 7fc2b6f4f640 1 -- 172.21.15.16:0/2482638787 shutdown_connections 2024-03-26T04:52:59.883 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.883+0000 7fc2b6f4f640 1 -- 172.21.15.16:0/2482638787 wait complete. 2024-03-26T04:52:59.883 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.884+0000 7fc2b6f4f640 1 Processor -- start 2024-03-26T04:52:59.883 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.884+0000 7fc2b6f4f640 1 -- start start 2024-03-26T04:52:59.884 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.884+0000 7fc2b6f4f640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc2b01a61d0 con 0x7fc2b0100530 2024-03-26T04:52:59.884 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.884+0000 7fc2b6f4f640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc2b01a73d0 con 0x7fc2b01091b0 2024-03-26T04:52:59.884 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.884+0000 7fc2b6f4f640 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7fc2b01a85d0 con 0x7fc2b010cd20 2024-03-26T04:52:59.884 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.884+0000 7fc2b54ec640 1 --1- >> v1:172.21.15.16:6790/0 conn(0x7fc2b010cd20 0x7fc2b01a48d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6790/0 says I am v1:172.21.15.16:56086/0 (socket says 172.21.15.16:56086) 2024-03-26T04:52:59.884 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.884+0000 7fc2b54ec640 1 -- 172.21.15.16:0/671986621 learned_addr learned my addr 172.21.15.16:0/671986621 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:52:59.884 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.884+0000 7fc2adffb640 1 -- 172.21.15.16:0/671986621 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 101252650 0 0) 0x7fc2b01a85d0 con 0x7fc2b010cd20 2024-03-26T04:52:59.884 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.884+0000 7fc2adffb640 1 -- 172.21.15.16:0/671986621 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2a0003520 con 0x7fc2b010cd20 2024-03-26T04:52:59.884 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.884+0000 7fc2adffb640 1 -- 172.21.15.16:0/671986621 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2641426204 0 0) 0x7fc2b01a61d0 con 0x7fc2b0100530 2024-03-26T04:52:59.884 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.884+0000 7fc2adffb640 1 -- 172.21.15.16:0/671986621 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2b01a85d0 con 0x7fc2b0100530 2024-03-26T04:52:59.884 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.885+0000 7fc2adffb640 1 -- 172.21.15.16:0/671986621 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3819031006 0 0) 0x7fc2b01a73d0 con 0x7fc2b01091b0 2024-03-26T04:52:59.884 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.885+0000 7fc2adffb640 1 -- 172.21.15.16:0/671986621 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2b01a61d0 con 0x7fc2b01091b0 2024-03-26T04:52:59.885 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.885+0000 7fc2adffb640 1 -- 172.21.15.16:0/671986621 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3212122258 0 0) 0x7fc2a0003520 con 0x7fc2b010cd20 2024-03-26T04:52:59.885 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.885+0000 7fc2adffb640 1 -- 172.21.15.16:0/671986621 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2b01a73d0 con 0x7fc2b010cd20 2024-03-26T04:52:59.885 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.885+0000 7fc2adffb640 1 -- 172.21.15.16:0/671986621 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fc2a8001c90 con 0x7fc2b010cd20 2024-03-26T04:52:59.885 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.885+0000 7fc2adffb640 1 -- 172.21.15.16:0/671986621 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 357967525 0 0) 0x7fc2b01a85d0 con 0x7fc2b0100530 2024-03-26T04:52:59.885 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.885+0000 7fc2adffb640 1 -- 172.21.15.16:0/671986621 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2a0003520 con 0x7fc2b0100530 2024-03-26T04:52:59.885 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.885+0000 7fc2adffb640 1 -- 172.21.15.16:0/671986621 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fc2a4004320 con 0x7fc2b0100530 2024-03-26T04:52:59.885 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.885+0000 7fc2adffb640 1 -- 172.21.15.16:0/671986621 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2592153457 0 0) 0x7fc2b01a61d0 con 0x7fc2b01091b0 2024-03-26T04:52:59.885 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.885+0000 7fc2adffb640 1 -- 172.21.15.16:0/671986621 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc2b01a85d0 con 0x7fc2b01091b0 2024-03-26T04:52:59.885 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.885+0000 7fc2adffb640 1 -- 172.21.15.16:0/671986621 <== mon.2 v1:172.21.15.16:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3418251452 0 0) 0x7fc2b01a73d0 con 0x7fc2b010cd20 2024-03-26T04:52:59.885 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.885+0000 7fc2adffb640 1 -- 172.21.15.16:0/671986621 >> v1:172.21.15.183:6789/0 conn(0x7fc2b01091b0 legacy=0x7fc2b01a10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:52:59.885 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.885+0000 7fc2adffb640 1 -- 172.21.15.16:0/671986621 >> v1:172.21.15.16:6789/0 conn(0x7fc2b0100530 legacy=0x7fc2b0105720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:52:59.885 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.885+0000 7fc2adffb640 1 -- 172.21.15.16:0/671986621 --> v1:172.21.15.16:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2b01a97d0 con 0x7fc2b010cd20 2024-03-26T04:52:59.886 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.885+0000 7fc2b6f4f640 1 -- 172.21.15.16:0/671986621 --> v1:172.21.15.16:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2b01a63c0 con 0x7fc2b010cd20 2024-03-26T04:52:59.886 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.885+0000 7fc2adffb640 1 -- 172.21.15.16:0/671986621 <== mon.2 v1:172.21.15.16:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7fc2a8003e60 con 0x7fc2b010cd20 2024-03-26T04:52:59.886 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.885+0000 7fc2adffb640 1 -- 172.21.15.16:0/671986621 <== mon.2 v1:172.21.15.16:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7fc2a8005f10 con 0x7fc2b010cd20 2024-03-26T04:52:59.886 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.885+0000 7fc2b6f4f640 1 -- 172.21.15.16:0/671986621 --> v1:172.21.15.16:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2b01a6850 con 0x7fc2b010cd20 2024-03-26T04:52:59.886 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.886+0000 7fc2b6f4f640 1 -- 172.21.15.16:0/671986621 --> v1:172.21.15.16:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2b0101a00 con 0x7fc2b010cd20 2024-03-26T04:52:59.886 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.886+0000 7fc2adffb640 1 -- 172.21.15.16:0/671986621 <== mon.2 v1:172.21.15.16:6790/0 7 ==== mgrmap(e 22) v1 ==== 91420+0+0 (unknown 3095259778 0 0) 0x7fc2a80039d0 con 0x7fc2b010cd20 2024-03-26T04:52:59.886 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.887+0000 7fc2adffb640 1 -- 172.21.15.16:0/671986621 <== mon.2 v1:172.21.15.16:6790/0 8 ==== osd_map(818..818 src has 253..818) v4 ==== 8186+0+0 (unknown 1225729805 0 0) 0x7fc2a80893e0 con 0x7fc2b010cd20 2024-03-26T04:52:59.886 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.887+0000 7fc2adffb640 1 -- 172.21.15.16:0/671986621 --> v1:172.21.15.16:6790/0 -- mon_subscribe({osdmap=819}) v3 -- 0x7fc2b01a73d0 con 0x7fc2b010cd20 2024-03-26T04:52:59.887 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:52:59.892+0000 7fc2adffb640 1 -- 172.21.15.16:0/671986621 <== mon.2 v1:172.21.15.16:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7fc2a8055f60 con 0x7fc2b010cd20 2024-03-26T04:53:00.037 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:59 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/614852167' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T04:53:00.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:59 smithi183 ceph-mon[33745]: osdmap e817: 8 total, 8 up, 8 in 2024-03-26T04:53:00.038 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:52:59 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/614852167' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T04:53:00.059 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:59 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/614852167' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T04:53:00.059 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:59 smithi016 ceph-mon[37215]: osdmap e817: 8 total, 8 up, 8 in 2024-03-26T04:53:00.059 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:52:59 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/614852167' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T04:53:00.059 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:59 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/614852167' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T04:53:00.059 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:59 smithi016 ceph-mon[29101]: osdmap e817: 8 total, 8 up, 8 in 2024-03-26T04:53:00.060 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:52:59 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/614852167' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "0"}]: dispatch 2024-03-26T04:53:00.072 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:00.076+0000 7fc2b6f4f640 1 -- 172.21.15.16:0/671986621 --> v1:172.21.15.16:6790/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "0"} v 0) v1 -- 0x7fc2b0104d50 con 0x7fc2b010cd20 2024-03-26T04:53:00.784 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:00.788+0000 7fc2adffb640 1 -- 172.21.15.16:0/671986621 <== mon.2 v1:172.21.15.16:6790/0 10 ==== osd_map(819..819 src has 253..819) v4 ==== 628+0+0 (unknown 459059167 0 0) 0x7fc2a8056890 con 0x7fc2b010cd20 2024-03-26T04:53:00.784 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:00.788+0000 7fc2adffb640 1 -- 172.21.15.16:0/671986621 --> v1:172.21.15.16:6790/0 -- mon_subscribe({osdmap=820}) v3 -- 0x7fc2a0003520 con 0x7fc2b010cd20 2024-03-26T04:53:00.788 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:00.792+0000 7fc2adffb640 1 -- 172.21.15.16:0/671986621 <== mon.2 v1:172.21.15.16:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool 9afc82fb-6d6b-4f10-901a-c3cc23ea025f v819) v1 ==== 221+0+0 (unknown 2131659651 0 0) 0x7fc2a80542d0 con 0x7fc2b010cd20 2024-03-26T04:53:00.904 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:00.908+0000 7fc2b6f4f640 1 -- 172.21.15.16:0/671986621 --> v1:172.21.15.16:6790/0 -- mon_command({"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "0"} v 0) v1 -- 0x7fc2b0112a70 con 0x7fc2b010cd20 2024-03-26T04:53:01.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:00 smithi016 ceph-mon[29101]: pgmap v1804: 188 pgs: 188 active+clean; 859 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:53:01.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:00 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/614852167' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T04:53:01.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:00 smithi016 ceph-mon[29101]: osdmap e818: 8 total, 8 up, 8 in 2024-03-26T04:53:01.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:00 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/671986621' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T04:53:01.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:00 smithi016 ceph-mon[29101]: from='client.57098 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T04:53:01.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:00 smithi016 ceph-mon[37215]: pgmap v1804: 188 pgs: 188 active+clean; 859 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:53:01.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:00 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/614852167' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T04:53:01.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:00 smithi016 ceph-mon[37215]: osdmap e818: 8 total, 8 up, 8 in 2024-03-26T04:53:01.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:00 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/671986621' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T04:53:01.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:00 smithi016 ceph-mon[37215]: from='client.57098 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T04:53:01.287 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:00 smithi183 ceph-mon[33745]: pgmap v1804: 188 pgs: 188 active+clean; 859 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:53:01.287 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:00 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/614852167' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_bytes", "val": "0"}]': finished 2024-03-26T04:53:01.287 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:00 smithi183 ceph-mon[33745]: osdmap e818: 8 total, 8 up, 8 in 2024-03-26T04:53:01.287 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:00 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/671986621' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T04:53:01.287 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:00 smithi183 ceph-mon[33745]: from='client.57098 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T04:53:01.799 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:01.803+0000 7fc2adffb640 1 -- 172.21.15.16:0/671986621 <== mon.2 v1:172.21.15.16:6790/0 12 ==== osd_map(820..820 src has 253..820) v4 ==== 628+0+0 (unknown 2786249825 0 0) 0x7fc2a805a2e0 con 0x7fc2b010cd20 2024-03-26T04:53:01.800 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:01.803+0000 7fc2adffb640 1 -- 172.21.15.16:0/671986621 --> v1:172.21.15.16:6790/0 -- mon_subscribe({osdmap=821}) v3 -- 0x7fc2b01a61d0 con 0x7fc2b010cd20 2024-03-26T04:53:01.804 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:01.808+0000 7fc2adffb640 1 -- 172.21.15.16:0/671986621 <== mon.2 v1:172.21.15.16:6790/0 13 ==== mon_command_ack([{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "0"}]=0 set-quota max_objects = 0 for pool 9afc82fb-6d6b-4f10-901a-c3cc23ea025f v820) v1 ==== 221+0+0 (unknown 4035454263 0 0) 0x7fc2a8055af0 con 0x7fc2b010cd20 2024-03-26T04:53:01.804 INFO:tasks.workunit.client.0.smithi016.stderr:set-quota max_objects = 0 for pool 9afc82fb-6d6b-4f10-901a-c3cc23ea025f 2024-03-26T04:53:01.808 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:01.811+0000 7fc2b6f4f640 1 -- 172.21.15.16:0/671986621 >> v1:172.21.15.16:6800/3868004154 conn(0x7fc2a006ea50 legacy=0x7fc2a0070f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:53:01.808 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:01.811+0000 7fc2b6f4f640 1 -- 172.21.15.16:0/671986621 >> v1:172.21.15.16:6790/0 conn(0x7fc2b010cd20 legacy=0x7fc2b01a48d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:53:01.808 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:01.812+0000 7fc2b6f4f640 1 -- 172.21.15.16:0/671986621 shutdown_connections 2024-03-26T04:53:01.808 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:01.812+0000 7fc2b6f4f640 1 -- 172.21.15.16:0/671986621 >> 172.21.15.16:0/671986621 conn(0x7fc2b00fbd90 msgr2=0x7fc2b006a2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:53:01.808 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:01.812+0000 7fc2b6f4f640 1 -- 172.21.15.16:0/671986621 shutdown_connections 2024-03-26T04:53:01.808 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:01.812+0000 7fc2b6f4f640 1 -- 172.21.15.16:0/671986621 wait complete. 2024-03-26T04:53:01.818 INFO:tasks.workunit.client.0.smithi016.stderr:+ sleep 30 2024-03-26T04:53:02.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:01 smithi016 ceph-mon[37215]: from='client.57098 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "0"}]': finished 2024-03-26T04:53:02.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:01 smithi016 ceph-mon[37215]: osdmap e819: 8 total, 8 up, 8 in 2024-03-26T04:53:02.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:01 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/671986621' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T04:53:02.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:01 smithi016 ceph-mon[37215]: from='client.57098 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T04:53:02.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:01 smithi016 ceph-mon[29101]: from='client.57098 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "0"}]': finished 2024-03-26T04:53:02.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:01 smithi016 ceph-mon[29101]: osdmap e819: 8 total, 8 up, 8 in 2024-03-26T04:53:02.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:01 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/671986621' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T04:53:02.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:01 smithi016 ceph-mon[29101]: from='client.57098 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T04:53:02.287 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:01 smithi183 ceph-mon[33745]: from='client.57098 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "0"}]': finished 2024-03-26T04:53:02.287 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:01 smithi183 ceph-mon[33745]: osdmap e819: 8 total, 8 up, 8 in 2024-03-26T04:53:02.287 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:01 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/671986621' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T04:53:02.287 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:01 smithi183 ceph-mon[33745]: from='client.57098 ' entity='client.admin' cmd=[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "0"}]: dispatch 2024-03-26T04:53:03.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:02 smithi016 ceph-mon[37215]: pgmap v1807: 188 pgs: 188 active+clean; 859 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:53:03.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:02 smithi016 ceph-mon[37215]: from='client.57098 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "0"}]': finished 2024-03-26T04:53:03.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:02 smithi016 ceph-mon[37215]: osdmap e820: 8 total, 8 up, 8 in 2024-03-26T04:53:03.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:02 smithi016 ceph-mon[29101]: pgmap v1807: 188 pgs: 188 active+clean; 859 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:53:03.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:02 smithi016 ceph-mon[29101]: from='client.57098 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "0"}]': finished 2024-03-26T04:53:03.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:02 smithi016 ceph-mon[29101]: osdmap e820: 8 total, 8 up, 8 in 2024-03-26T04:53:03.290 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:02 smithi183 ceph-mon[33745]: pgmap v1807: 188 pgs: 188 active+clean; 859 KiB data, 895 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:53:03.290 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:02 smithi183 ceph-mon[33745]: from='client.57098 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "field": "max_objects", "val": "0"}]': finished 2024-03-26T04:53:03.291 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:02 smithi183 ceph-mon[33745]: osdmap e820: 8 total, 8 up, 8 in 2024-03-26T04:53:04.188 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:53:03 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:53:05.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:04 smithi016 ceph-mon[37215]: pgmap v1809: 188 pgs: 188 active+clean; 871 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 2.4 KiB/s wr, 1 op/s 2024-03-26T04:53:05.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:04 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:53:05.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:04 smithi016 ceph-mon[29101]: pgmap v1809: 188 pgs: 188 active+clean; 871 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 2.4 KiB/s wr, 1 op/s 2024-03-26T04:53:05.159 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:04 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:53:05.287 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:04 smithi183 ceph-mon[33745]: pgmap v1809: 188 pgs: 188 active+clean; 871 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 2.4 KiB/s wr, 1 op/s 2024-03-26T04:53:05.287 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:04 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:53:07.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:06 smithi016 ceph-mon[37215]: pgmap v1810: 188 pgs: 188 active+clean; 871 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 2.0 KiB/s wr, 0 op/s 2024-03-26T04:53:07.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:06 smithi016 ceph-mon[29101]: pgmap v1810: 188 pgs: 188 active+clean; 871 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 2.0 KiB/s wr, 0 op/s 2024-03-26T04:53:07.287 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:06 smithi183 ceph-mon[33745]: pgmap v1810: 188 pgs: 188 active+clean; 871 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 2.0 KiB/s wr, 0 op/s 2024-03-26T04:53:09.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:08 smithi016 ceph-mon[37215]: pgmap v1811: 188 pgs: 188 active+clean; 871 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 1 op/s 2024-03-26T04:53:09.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:08 smithi016 ceph-mon[29101]: pgmap v1811: 188 pgs: 188 active+clean; 871 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 1 op/s 2024-03-26T04:53:09.159 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:53:08 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:53:08] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:53:09.287 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:08 smithi183 ceph-mon[33745]: pgmap v1811: 188 pgs: 188 active+clean; 871 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.5 KiB/s wr, 1 op/s 2024-03-26T04:53:10.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:09 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:53:10.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:09 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:53:10.287 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:09 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:53:11.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:10 smithi016 ceph-mon[37215]: pgmap v1812: 188 pgs: 188 active+clean; 871 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 1 op/s 2024-03-26T04:53:11.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:10 smithi016 ceph-mon[29101]: pgmap v1812: 188 pgs: 188 active+clean; 871 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 1 op/s 2024-03-26T04:53:11.287 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:10 smithi183 ceph-mon[33745]: pgmap v1812: 188 pgs: 188 active+clean; 871 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.3 KiB/s wr, 1 op/s 2024-03-26T04:53:13.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:12 smithi016 ceph-mon[37215]: pgmap v1813: 188 pgs: 188 active+clean; 871 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 1.2 KiB/s wr, 1 op/s 2024-03-26T04:53:13.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:12 smithi016 ceph-mon[29101]: pgmap v1813: 188 pgs: 188 active+clean; 871 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 1.2 KiB/s wr, 1 op/s 2024-03-26T04:53:13.287 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:12 smithi183 ceph-mon[33745]: pgmap v1813: 188 pgs: 188 active+clean; 871 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 1.2 KiB/s wr, 1 op/s 2024-03-26T04:53:14.287 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:53:13 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:53:15.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:14 smithi016 ceph-mon[37215]: pgmap v1814: 188 pgs: 188 active+clean; 871 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.0 KiB/s wr, 1 op/s 2024-03-26T04:53:15.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:14 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:53:15.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:14 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:53:15.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:14 smithi016 ceph-mon[29101]: pgmap v1814: 188 pgs: 188 active+clean; 871 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.0 KiB/s wr, 1 op/s 2024-03-26T04:53:15.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:14 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:53:15.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:14 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:53:15.287 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:14 smithi183 ceph-mon[33745]: pgmap v1814: 188 pgs: 188 active+clean; 871 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 1.3 KiB/s rd, 1.0 KiB/s wr, 1 op/s 2024-03-26T04:53:15.287 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:14 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:53:15.287 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:14 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-26T04:53:16.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:15 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:53:16.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:15 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:53:16.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:15 smithi016 ceph-mon[29101]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:53:16.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:15 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:53:16.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:15 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:53:16.159 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:15 smithi016 ceph-mon[37215]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:53:16.287 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:15 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-26T04:53:16.287 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:15 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-26T04:53:16.287 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:15 smithi183 ceph-mon[33745]: from='mgr.14616 ' entity='mgr.y' 2024-03-26T04:53:17.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:16 smithi016 ceph-mon[37215]: pgmap v1815: 188 pgs: 188 active+clean; 871 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:53:17.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:16 smithi016 ceph-mon[29101]: pgmap v1815: 188 pgs: 188 active+clean; 871 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:53:17.287 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:16 smithi183 ceph-mon[33745]: pgmap v1815: 188 pgs: 188 active+clean; 871 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:53:19.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:18 smithi016 ceph-mon[37215]: pgmap v1816: 188 pgs: 188 active+clean; 871 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:53:19.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:18 smithi016 ceph-mon[29101]: pgmap v1816: 188 pgs: 188 active+clean; 871 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:53:19.158 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:53:18 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:53:18] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:53:19.287 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:18 smithi183 ceph-mon[33745]: pgmap v1816: 188 pgs: 188 active+clean; 871 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:53:21.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:20 smithi016 ceph-mon[37215]: pgmap v1817: 188 pgs: 188 active+clean; 871 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:53:21.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:20 smithi016 ceph-mon[29101]: pgmap v1817: 188 pgs: 188 active+clean; 871 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:53:21.287 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:20 smithi183 ceph-mon[33745]: pgmap v1817: 188 pgs: 188 active+clean; 871 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:53:23.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:22 smithi016 ceph-mon[37215]: pgmap v1818: 188 pgs: 188 active+clean; 871 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:53:23.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:22 smithi016 ceph-mon[29101]: pgmap v1818: 188 pgs: 188 active+clean; 871 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:53:23.287 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:22 smithi183 ceph-mon[33745]: pgmap v1818: 188 pgs: 188 active+clean; 871 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:53:24.287 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:53:23 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:53:25.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:24 smithi016 ceph-mon[37215]: pgmap v1819: 188 pgs: 188 active+clean; 871 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:53:25.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:24 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:53:25.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:24 smithi016 ceph-mon[37215]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:53:25.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:24 smithi016 ceph-mon[29101]: pgmap v1819: 188 pgs: 188 active+clean; 871 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:53:25.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:24 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:53:25.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:24 smithi016 ceph-mon[29101]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:53:25.287 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:24 smithi183 ceph-mon[33745]: pgmap v1819: 188 pgs: 188 active+clean; 871 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:53:25.287 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:24 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:53:25.287 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:24 smithi183 ceph-mon[33745]: from='mgr.14616 v1:172.21.15.16:0/1099951999' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-26T04:53:27.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:26 smithi016 ceph-mon[29101]: pgmap v1820: 188 pgs: 188 active+clean; 871 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:53:27.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:26 smithi016 ceph-mon[37215]: pgmap v1820: 188 pgs: 188 active+clean; 871 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:53:27.287 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:26 smithi183 ceph-mon[33745]: pgmap v1820: 188 pgs: 188 active+clean; 871 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s 2024-03-26T04:53:29.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:28 smithi016 ceph-mon[29101]: pgmap v1821: 188 pgs: 188 active+clean; 871 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:53:29.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:28 smithi016 ceph-mon[37215]: pgmap v1821: 188 pgs: 188 active+clean; 871 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:53:29.158 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:53:28 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:53:28] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:53:29.287 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:28 smithi183 ceph-mon[33745]: pgmap v1821: 188 pgs: 188 active+clean; 871 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-26T04:53:31.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:30 smithi016 ceph-mon[29101]: pgmap v1822: 188 pgs: 188 active+clean; 871 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:53:31.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:30 smithi016 ceph-mon[37215]: pgmap v1822: 188 pgs: 188 active+clean; 871 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:53:31.287 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:30 smithi183 ceph-mon[33745]: pgmap v1822: 188 pgs: 188 active+clean; 871 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:53:31.820 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph osd pool delete 9afc82fb-6d6b-4f10-901a-c3cc23ea025f 9afc82fb-6d6b-4f10-901a-c3cc23ea025f --yes-i-really-really-mean-it 2024-03-26T04:53:31.914 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:31.918+0000 7f7d9e458640 1 -- 172.21.15.16:0/1614283377 >> v1:172.21.15.16:6789/0 conn(0x7f7d9810c4f0 legacy=0x7f7d9810e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:53:31.914 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:31.918+0000 7f7d9e458640 1 -- 172.21.15.16:0/1614283377 shutdown_connections 2024-03-26T04:53:31.914 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:31.918+0000 7f7d9e458640 1 -- 172.21.15.16:0/1614283377 >> 172.21.15.16:0/1614283377 conn(0x7f7d98075610 msgr2=0x7f7d98075a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:53:31.914 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:31.918+0000 7f7d9e458640 1 -- 172.21.15.16:0/1614283377 shutdown_connections 2024-03-26T04:53:31.915 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:31.918+0000 7f7d9e458640 1 -- 172.21.15.16:0/1614283377 wait complete. 2024-03-26T04:53:31.915 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:31.919+0000 7f7d9e458640 1 Processor -- start 2024-03-26T04:53:31.915 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:31.919+0000 7f7d9e458640 1 -- start start 2024-03-26T04:53:31.917 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:31.919+0000 7f7d9e458640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7d981a6340 con 0x7f7d98108920 2024-03-26T04:53:31.917 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:31.919+0000 7f7d9e458640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7d981a7540 con 0x7f7d9810c4f0 2024-03-26T04:53:31.917 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:31.919+0000 7f7d9e458640 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f7d981a8740 con 0x7f7d98105a80 2024-03-26T04:53:31.917 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:31.919+0000 7f7d97fff640 1 --1- >> v1:172.21.15.16:6790/0 conn(0x7f7d98105a80 0x7f7d980fef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6790/0 says I am v1:172.21.15.16:41572/0 (socket says 172.21.15.16:41572) 2024-03-26T04:53:31.918 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:31.919+0000 7f7d97fff640 1 -- 172.21.15.16:0/688933399 learned_addr learned my addr 172.21.15.16:0/688933399 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:53:31.918 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:31.919+0000 7f7d957fa640 1 -- 172.21.15.16:0/688933399 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3685071449 0 0) 0x7f7d981a8740 con 0x7f7d98105a80 2024-03-26T04:53:31.918 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:31.919+0000 7f7d957fa640 1 -- 172.21.15.16:0/688933399 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d84004630 con 0x7f7d98105a80 2024-03-26T04:53:31.918 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:31.920+0000 7f7d957fa640 1 -- 172.21.15.16:0/688933399 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3181638910 0 0) 0x7f7d981a6340 con 0x7f7d98108920 2024-03-26T04:53:31.918 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:31.920+0000 7f7d957fa640 1 -- 172.21.15.16:0/688933399 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d981a8740 con 0x7f7d98108920 2024-03-26T04:53:31.918 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:31.920+0000 7f7d957fa640 1 -- 172.21.15.16:0/688933399 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1438952836 0 0) 0x7f7d981a7540 con 0x7f7d9810c4f0 2024-03-26T04:53:31.918 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:31.920+0000 7f7d957fa640 1 -- 172.21.15.16:0/688933399 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d981a6340 con 0x7f7d9810c4f0 2024-03-26T04:53:31.918 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:31.920+0000 7f7d957fa640 1 -- 172.21.15.16:0/688933399 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4215038771 0 0) 0x7f7d84004630 con 0x7f7d98105a80 2024-03-26T04:53:31.918 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:31.920+0000 7f7d957fa640 1 -- 172.21.15.16:0/688933399 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d981a7540 con 0x7f7d98105a80 2024-03-26T04:53:31.918 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:31.920+0000 7f7d957fa640 1 -- 172.21.15.16:0/688933399 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 671580383 0 0) 0x7f7d981a8740 con 0x7f7d98108920 2024-03-26T04:53:31.918 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:31.920+0000 7f7d957fa640 1 -- 172.21.15.16:0/688933399 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d84004630 con 0x7f7d98108920 2024-03-26T04:53:31.919 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:31.920+0000 7f7d957fa640 1 -- 172.21.15.16:0/688933399 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f7d880012d0 con 0x7f7d98105a80 2024-03-26T04:53:31.919 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:31.920+0000 7f7d957fa640 1 -- 172.21.15.16:0/688933399 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f7d800018b0 con 0x7f7d98108920 2024-03-26T04:53:31.919 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:31.920+0000 7f7d957fa640 1 -- 172.21.15.16:0/688933399 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3665529291 0 0) 0x7f7d981a6340 con 0x7f7d9810c4f0 2024-03-26T04:53:31.919 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:31.920+0000 7f7d957fa640 1 -- 172.21.15.16:0/688933399 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d981a8740 con 0x7f7d9810c4f0 2024-03-26T04:53:31.919 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:31.920+0000 7f7d957fa640 1 -- 172.21.15.16:0/688933399 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f7d90004610 con 0x7f7d9810c4f0 2024-03-26T04:53:31.919 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:31.920+0000 7f7d957fa640 1 -- 172.21.15.16:0/688933399 <== mon.2 v1:172.21.15.16:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 917273893 0 0) 0x7f7d981a7540 con 0x7f7d98105a80 2024-03-26T04:53:31.919 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:31.920+0000 7f7d957fa640 1 -- 172.21.15.16:0/688933399 >> v1:172.21.15.183:6789/0 conn(0x7f7d9810c4f0 legacy=0x7f7d981a4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:53:31.919 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:31.920+0000 7f7d957fa640 1 -- 172.21.15.16:0/688933399 >> v1:172.21.15.16:6789/0 conn(0x7f7d98108920 legacy=0x7f7d980ff630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:53:31.919 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:31.920+0000 7f7d957fa640 1 -- 172.21.15.16:0/688933399 --> v1:172.21.15.16:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d981a9940 con 0x7f7d98105a80 2024-03-26T04:53:31.919 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:31.920+0000 7f7d9e458640 1 -- 172.21.15.16:0/688933399 --> v1:172.21.15.16:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d981a6530 con 0x7f7d98105a80 2024-03-26T04:53:31.919 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:31.920+0000 7f7d9e458640 1 -- 172.21.15.16:0/688933399 --> v1:172.21.15.16:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d981a69c0 con 0x7f7d98105a80 2024-03-26T04:53:31.919 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:31.920+0000 7f7d957fa640 1 -- 172.21.15.16:0/688933399 <== mon.2 v1:172.21.15.16:6790/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f7d880015e0 con 0x7f7d98105a80 2024-03-26T04:53:31.919 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:31.920+0000 7f7d957fa640 1 -- 172.21.15.16:0/688933399 <== mon.2 v1:172.21.15.16:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f7d88005b50 con 0x7f7d98105a80 2024-03-26T04:53:31.920 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:31.921+0000 7f7d9e458640 1 -- 172.21.15.16:0/688933399 --> v1:172.21.15.16:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d78005080 con 0x7f7d98105a80 2024-03-26T04:53:31.920 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:31.922+0000 7f7d957fa640 1 -- 172.21.15.16:0/688933399 <== mon.2 v1:172.21.15.16:6790/0 7 ==== mgrmap(e 22) v1 ==== 91420+0+0 (unknown 3095259778 0 0) 0x7f7d880038a0 con 0x7f7d98105a80 2024-03-26T04:53:31.920 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:31.922+0000 7f7d957fa640 1 -- 172.21.15.16:0/688933399 <== mon.2 v1:172.21.15.16:6790/0 8 ==== osd_map(820..820 src has 253..820) v4 ==== 8186+0+0 (unknown 250744155 0 0) 0x7f7d88088f70 con 0x7f7d98105a80 2024-03-26T04:53:31.920 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:31.922+0000 7f7d957fa640 1 -- 172.21.15.16:0/688933399 --> v1:172.21.15.16:6790/0 -- mon_subscribe({osdmap=821}) v3 -- 0x7f7d981a7540 con 0x7f7d98105a80 2024-03-26T04:53:31.922 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:31.927+0000 7f7d957fa640 1 -- 172.21.15.16:0/688933399 <== mon.2 v1:172.21.15.16:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f7d88055af0 con 0x7f7d98105a80 2024-03-26T04:53:32.107 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:32.112+0000 7f7d9e458640 1 -- 172.21.15.16:0/688933399 --> v1:172.21.15.16:6790/0 -- mon_command({"prefix": "osd pool delete", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "pool2": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7d78005330 con 0x7f7d98105a80 2024-03-26T04:53:32.844 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:32.848+0000 7f7d957fa640 1 -- 172.21.15.16:0/688933399 <== mon.2 v1:172.21.15.16:6790/0 10 ==== osd_map(821..821 src has 253..821) v4 ==== 296+0+0 (unknown 1922079742 0 0) 0x7f7d88056420 con 0x7f7d98105a80 2024-03-26T04:53:32.845 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:32.848+0000 7f7d957fa640 1 -- 172.21.15.16:0/688933399 --> v1:172.21.15.16:6790/0 -- mon_subscribe({osdmap=822}) v3 -- 0x7f7d84004630 con 0x7f7d98105a80 2024-03-26T04:53:32.849 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:32.853+0000 7f7d957fa640 1 -- 172.21.15.16:0/688933399 <== mon.2 v1:172.21.15.16:6790/0 11 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "pool2": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "yes_i_really_really_mean_it": true}]=0 pool '9afc82fb-6d6b-4f10-901a-c3cc23ea025f' removed v821) v1 ==== 248+0+0 (unknown 2971996417 0 0) 0x7f7d88053e60 con 0x7f7d98105a80 2024-03-26T04:53:32.966 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:32.970+0000 7f7d9e458640 1 -- 172.21.15.16:0/688933399 --> v1:172.21.15.16:6790/0 -- mon_command({"prefix": "osd pool delete", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "pool2": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7d78001140 con 0x7f7d98105a80 2024-03-26T04:53:32.967 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:32.971+0000 7f7d957fa640 1 -- 172.21.15.16:0/688933399 <== mon.2 v1:172.21.15.16:6790/0 12 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "pool2": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "yes_i_really_really_mean_it": true}]=0 pool '9afc82fb-6d6b-4f10-901a-c3cc23ea025f' does not exist v821) v1 ==== 255+0+0 (unknown 2805353675 0 0) 0x7f7d88055680 con 0x7f7d98105a80 2024-03-26T04:53:32.967 INFO:tasks.workunit.client.0.smithi016.stderr:pool '9afc82fb-6d6b-4f10-901a-c3cc23ea025f' does not exist 2024-03-26T04:53:32.969 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:32.973+0000 7f7d9e458640 1 -- 172.21.15.16:0/688933399 >> v1:172.21.15.16:6800/3868004154 conn(0x7f7d8406eb70 legacy=0x7f7d84071030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:53:32.969 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:32.973+0000 7f7d9e458640 1 -- 172.21.15.16:0/688933399 >> v1:172.21.15.16:6790/0 conn(0x7f7d98105a80 legacy=0x7f7d980fef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:53:32.969 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:32.974+0000 7f7d9e458640 1 -- 172.21.15.16:0/688933399 shutdown_connections 2024-03-26T04:53:32.969 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:32.974+0000 7f7d9e458640 1 -- 172.21.15.16:0/688933399 >> 172.21.15.16:0/688933399 conn(0x7f7d98075610 msgr2=0x7f7d98112820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:53:32.970 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:32.974+0000 7f7d9e458640 1 -- 172.21.15.16:0/688933399 shutdown_connections 2024-03-26T04:53:32.970 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:32.974+0000 7f7d9e458640 1 -- 172.21.15.16:0/688933399 wait complete. 2024-03-26T04:53:32.978 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph osd pool delete dcd349bf-3588-4b0c-9f5d-114ed7e44baf dcd349bf-3588-4b0c-9f5d-114ed7e44baf --yes-i-really-really-mean-it 2024-03-26T04:53:33.061 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.065+0000 7f5b8d0fb640 1 -- 172.21.15.16:0/680187986 >> v1:172.21.15.16:6790/0 conn(0x7f5b881091c0 legacy=0x7f5b8810b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:53:33.061 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.065+0000 7f5b8d0fb640 1 -- 172.21.15.16:0/680187986 shutdown_connections 2024-03-26T04:53:33.061 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.065+0000 7f5b8d0fb640 1 -- 172.21.15.16:0/680187986 >> 172.21.15.16:0/680187986 conn(0x7f5b88076fa0 msgr2=0x7f5b880fe300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:53:33.061 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.065+0000 7f5b8d0fb640 1 -- 172.21.15.16:0/680187986 shutdown_connections 2024-03-26T04:53:33.061 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.065+0000 7f5b8d0fb640 1 -- 172.21.15.16:0/680187986 wait complete. 2024-03-26T04:53:33.061 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.066+0000 7f5b8d0fb640 1 Processor -- start 2024-03-26T04:53:33.062 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.066+0000 7f5b8d0fb640 1 -- start start 2024-03-26T04:53:33.064 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.066+0000 7f5b8d0fb640 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5b88105cb0 con 0x7f5b881091c0 2024-03-26T04:53:33.065 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.066+0000 7f5b8d0fb640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5b881a7810 con 0x7f5b88100540 2024-03-26T04:53:33.065 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.066+0000 7f5b8d0fb640 1 -- --> v1:172.21.15.16:6790/0 -- auth(proto 0 34 bytes epoch 0) v1 -- 0x7f5b881a89b0 con 0x7f5b8810cd30 2024-03-26T04:53:33.065 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.066+0000 7f5b7ffff640 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5b881091c0 0x7f5b88104230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33002/0 (socket says 172.21.15.16:33002) 2024-03-26T04:53:33.065 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.066+0000 7f5b7ffff640 1 -- 172.21.15.16:0/3203726836 learned_addr learned my addr 172.21.15.16:0/3203726836 (peer_addr_for_me v1:172.21.15.16:0/0) 2024-03-26T04:53:33.065 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.066+0000 7f5b8759e640 1 --1- >> v1:172.21.15.16:6790/0 conn(0x7f5b8810cd30 0x7f5b881a50e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6790/0 says I am v1:172.21.15.16:41602/0 (socket says 172.21.15.16:41602) 2024-03-26T04:53:33.065 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.066+0000 7f5b84d99640 1 -- 172.21.15.16:0/3203726836 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4172800422 0 0) 0x7f5b88105cb0 con 0x7f5b881091c0 2024-03-26T04:53:33.065 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.066+0000 7f5b84d99640 1 -- 172.21.15.16:0/3203726836 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b74003520 con 0x7f5b881091c0 2024-03-26T04:53:33.065 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.066+0000 7f5b84d99640 1 -- 172.21.15.16:0/3203726836 <== mon.2 v1:172.21.15.16:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 400197884 0 0) 0x7f5b881a89b0 con 0x7f5b8810cd30 2024-03-26T04:53:33.065 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.066+0000 7f5b84d99640 1 -- 172.21.15.16:0/3203726836 --> v1:172.21.15.16:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b88105cb0 con 0x7f5b8810cd30 2024-03-26T04:53:33.065 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.066+0000 7f5b84d99640 1 -- 172.21.15.16:0/3203726836 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4065664602 0 0) 0x7f5b881a7810 con 0x7f5b88100540 2024-03-26T04:53:33.065 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.066+0000 7f5b84d99640 1 -- 172.21.15.16:0/3203726836 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b881a89b0 con 0x7f5b88100540 2024-03-26T04:53:33.065 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.066+0000 7f5b84d99640 1 -- 172.21.15.16:0/3203726836 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1436437996 0 0) 0x7f5b74003520 con 0x7f5b881091c0 2024-03-26T04:53:33.065 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.066+0000 7f5b84d99640 1 -- 172.21.15.16:0/3203726836 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b881a7810 con 0x7f5b881091c0 2024-03-26T04:53:33.066 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.066+0000 7f5b84d99640 1 -- 172.21.15.16:0/3203726836 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f5b70004480 con 0x7f5b881091c0 2024-03-26T04:53:33.066 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.066+0000 7f5b84d99640 1 -- 172.21.15.16:0/3203726836 <== mon.2 v1:172.21.15.16:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2917580118 0 0) 0x7f5b88105cb0 con 0x7f5b8810cd30 2024-03-26T04:53:33.066 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.067+0000 7f5b84d99640 1 -- 172.21.15.16:0/3203726836 --> v1:172.21.15.16:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b74003520 con 0x7f5b8810cd30 2024-03-26T04:53:33.066 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.067+0000 7f5b84d99640 1 -- 172.21.15.16:0/3203726836 <== mon.2 v1:172.21.15.16:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f5b80001bd0 con 0x7f5b8810cd30 2024-03-26T04:53:33.066 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.067+0000 7f5b84d99640 1 -- 172.21.15.16:0/3203726836 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3728997189 0 0) 0x7f5b881a89b0 con 0x7f5b88100540 2024-03-26T04:53:33.066 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.067+0000 7f5b84d99640 1 -- 172.21.15.16:0/3203726836 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b88105cb0 con 0x7f5b88100540 2024-03-26T04:53:33.066 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.067+0000 7f5b84d99640 1 -- 172.21.15.16:0/3203726836 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f5b78001830 con 0x7f5b88100540 2024-03-26T04:53:33.066 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.067+0000 7f5b84d99640 1 -- 172.21.15.16:0/3203726836 <== mon.0 v1:172.21.15.16:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2418117539 0 0) 0x7f5b881a7810 con 0x7f5b881091c0 2024-03-26T04:53:33.066 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.067+0000 7f5b84d99640 1 -- 172.21.15.16:0/3203726836 >> v1:172.21.15.16:6790/0 conn(0x7f5b8810cd30 legacy=0x7f5b881a50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:53:33.066 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.067+0000 7f5b84d99640 1 -- 172.21.15.16:0/3203726836 >> v1:172.21.15.183:6789/0 conn(0x7f5b88100540 legacy=0x7f5b88103b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:53:33.066 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.067+0000 7f5b84d99640 1 -- 172.21.15.16:0/3203726836 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b881a9b50 con 0x7f5b881091c0 2024-03-26T04:53:33.066 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.067+0000 7f5b8d0fb640 1 -- 172.21.15.16:0/3203726836 --> v1:172.21.15.16:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b881a8ba0 con 0x7f5b881091c0 2024-03-26T04:53:33.066 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.067+0000 7f5b8d0fb640 1 -- 172.21.15.16:0/3203726836 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b881a9030 con 0x7f5b881091c0 2024-03-26T04:53:33.066 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.067+0000 7f5b84d99640 1 -- 172.21.15.16:0/3203726836 <== mon.0 v1:172.21.15.16:6789/0 5 ==== config(25 keys) v1 ==== 1068+0+0 (unknown 2750266749 0 0) 0x7f5b70001aa0 con 0x7f5b881091c0 2024-03-26T04:53:33.067 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.067+0000 7f5b84d99640 1 -- 172.21.15.16:0/3203726836 <== mon.0 v1:172.21.15.16:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1455445705 0 0) 0x7f5b70004e40 con 0x7f5b881091c0 2024-03-26T04:53:33.067 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.068+0000 7f5b8d0fb640 1 -- 172.21.15.16:0/3203726836 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b68005080 con 0x7f5b881091c0 2024-03-26T04:53:33.067 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.068+0000 7f5b84d99640 1 -- 172.21.15.16:0/3203726836 <== mon.0 v1:172.21.15.16:6789/0 7 ==== mgrmap(e 22) v1 ==== 91420+0+0 (unknown 3095259778 0 0) 0x7f5b70004fa0 con 0x7f5b881091c0 2024-03-26T04:53:33.067 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.069+0000 7f5b84d99640 1 -- 172.21.15.16:0/3203726836 <== mon.0 v1:172.21.15.16:6789/0 8 ==== osd_map(821..821 src has 253..821) v4 ==== 7798+0+0 (unknown 98018064 0 0) 0x7f5b700890e0 con 0x7f5b881091c0 2024-03-26T04:53:33.067 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.069+0000 7f5b84d99640 1 -- 172.21.15.16:0/3203726836 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=822}) v3 -- 0x7f5b881a7810 con 0x7f5b881091c0 2024-03-26T04:53:33.069 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.073+0000 7f5b84d99640 1 -- 172.21.15.16:0/3203726836 <== mon.0 v1:172.21.15.16:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184932 (unknown 1092875540 0 686004322) 0x7f5b70055de0 con 0x7f5b881091c0 2024-03-26T04:53:33.157 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:32 smithi016 ceph-mon[37215]: pgmap v1823: 188 pgs: 188 active+clean; 871 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:53:33.157 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:32 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/688933399' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "pool2": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:53:33.158 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:32 smithi016 ceph-mon[37215]: from='client.57104 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "pool2": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:53:33.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:32 smithi016 ceph-mon[29101]: pgmap v1823: 188 pgs: 188 active+clean; 871 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:53:33.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:32 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/688933399' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "pool2": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:53:33.158 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:32 smithi016 ceph-mon[29101]: from='client.57104 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "pool2": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:53:33.254 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.259+0000 7f5b8d0fb640 1 -- 172.21.15.16:0/3203726836 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "pool2": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5b68005330 con 0x7f5b881091c0 2024-03-26T04:53:33.287 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:32 smithi183 ceph-mon[33745]: pgmap v1823: 188 pgs: 188 active+clean; 871 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-26T04:53:33.287 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:32 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/688933399' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "pool2": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:53:33.287 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:32 smithi183 ceph-mon[33745]: from='client.57104 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "pool2": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:53:33.859 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.863+0000 7f5b84d99640 1 -- 172.21.15.16:0/3203726836 <== mon.0 v1:172.21.15.16:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "pool2": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "yes_i_really_really_mean_it": true}]=0 pool 'dcd349bf-3588-4b0c-9f5d-114ed7e44baf' removed v822) v1 ==== 248+0+0 (unknown 33523026 0 0) 0x7f5b70054150 con 0x7f5b881091c0 2024-03-26T04:53:33.859 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.863+0000 7f5b84d99640 1 -- 172.21.15.16:0/3203726836 <== mon.0 v1:172.21.15.16:6789/0 11 ==== osd_map(822..822 src has 253..822) v4 ==== 296+0+0 (unknown 3038599798 0 0) 0x7f5b70056710 con 0x7f5b881091c0 2024-03-26T04:53:33.859 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.864+0000 7f5b84d99640 1 -- 172.21.15.16:0/3203726836 --> v1:172.21.15.16:6789/0 -- mon_subscribe({osdmap=823}) v3 -- 0x7f5b881a89b0 con 0x7f5b881091c0 2024-03-26T04:53:33.982 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.986+0000 7f5b8d0fb640 1 -- 172.21.15.16:0/3203726836 --> v1:172.21.15.16:6789/0 -- mon_command({"prefix": "osd pool delete", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "pool2": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5b68001180 con 0x7f5b881091c0 2024-03-26T04:53:33.982 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.986+0000 7f5b84d99640 1 -- 172.21.15.16:0/3203726836 <== mon.0 v1:172.21.15.16:6789/0 12 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "pool2": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "yes_i_really_really_mean_it": true}]=0 pool 'dcd349bf-3588-4b0c-9f5d-114ed7e44baf' does not exist v822) v1 ==== 255+0+0 (unknown 1619866721 0 0) 0x7f5b70055970 con 0x7f5b881091c0 2024-03-26T04:53:33.982 INFO:tasks.workunit.client.0.smithi016.stderr:pool 'dcd349bf-3588-4b0c-9f5d-114ed7e44baf' does not exist 2024-03-26T04:53:33.984 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.988+0000 7f5b8d0fb640 1 -- 172.21.15.16:0/3203726836 >> v1:172.21.15.16:6800/3868004154 conn(0x7f5b7406ea30 legacy=0x7f5b74070ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:53:33.984 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.988+0000 7f5b8d0fb640 1 -- 172.21.15.16:0/3203726836 >> v1:172.21.15.16:6789/0 conn(0x7f5b881091c0 legacy=0x7f5b88104230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-26T04:53:33.984 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.988+0000 7f5b8d0fb640 1 -- 172.21.15.16:0/3203726836 shutdown_connections 2024-03-26T04:53:33.984 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.988+0000 7f5b8d0fb640 1 -- 172.21.15.16:0/3203726836 >> 172.21.15.16:0/3203726836 conn(0x7f5b88076fa0 msgr2=0x7f5b8810f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-26T04:53:33.985 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.989+0000 7f5b8d0fb640 1 -- 172.21.15.16:0/3203726836 shutdown_connections 2024-03-26T04:53:33.985 INFO:tasks.workunit.client.0.smithi016.stderr:2024-03-26T04:53:33.989+0000 7f5b8d0fb640 1 -- 172.21.15.16:0/3203726836 wait complete. 2024-03-26T04:53:33.993 INFO:tasks.workunit.client.0.smithi016.stdout:OK 2024-03-26T04:53:33.994 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo OK 2024-03-26T04:53:33.995 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-26T04:53:33.995 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-26T04:53:34.064 INFO:tasks.workunit:Stopping ['rados/test.sh', 'rados/test_pool_quota.sh'] on client.0... 2024-03-26T04:53:34.064 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-26T04:53:34.128 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:33 smithi016 ceph-mon[29101]: from='client.57104 ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "pool2": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T04:53:34.128 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:33 smithi016 ceph-mon[29101]: osdmap e821: 8 total, 8 up, 8 in 2024-03-26T04:53:34.128 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:33 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/688933399' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "pool2": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:53:34.128 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:33 smithi016 ceph-mon[29101]: from='client.57104 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "pool2": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:53:34.128 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:33 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3203726836' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "pool2": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:53:34.129 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:33 smithi016 ceph-mon[37215]: from='client.57104 ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "pool2": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T04:53:34.129 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:33 smithi016 ceph-mon[37215]: osdmap e821: 8 total, 8 up, 8 in 2024-03-26T04:53:34.129 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:33 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/688933399' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "pool2": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:53:34.129 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:33 smithi016 ceph-mon[37215]: from='client.57104 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "pool2": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:53:34.129 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:33 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3203726836' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "pool2": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:53:34.188 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:53:33 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug there is no tcmu-runner data available 2024-03-26T04:53:34.189 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:33 smithi183 ceph-mon[33745]: from='client.57104 ' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "pool2": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T04:53:34.189 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:33 smithi183 ceph-mon[33745]: osdmap e821: 8 total, 8 up, 8 in 2024-03-26T04:53:34.189 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:33 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/688933399' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "pool2": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:53:34.189 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:33 smithi183 ceph-mon[33745]: from='client.57104 ' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "pool2": "9afc82fb-6d6b-4f10-901a-c3cc23ea025f", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:53:34.189 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:33 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3203726836' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "pool2": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:53:34.400 DEBUG:teuthology.parallel:result is None 2024-03-26T04:53:34.401 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-26T04:53:34.467 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-26T04:53:34.467 DEBUG:teuthology.orchestra.run.smithi016:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-03-26T04:53:34.528 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-03-26T04:53:34.528 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-26T04:53:34.540 INFO:tasks.cephadm:Teardown begin 2024-03-26T04:53:34.540 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T04:53:34.595 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T04:53:34.630 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-26T04:53:34.631 DEBUG:teuthology.orchestra.run.smithi016:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-26T04:53:34.650 DEBUG:teuthology.orchestra.run.smithi183:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-26T04:53:34.688 INFO:tasks.cephadm:Stopping all daemons... 2024-03-26T04:53:34.688 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-26T04:53:34.689 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-d7d59826-eb26-11ee-95d0-87774f69a715@mon.a 2024-03-26T04:53:35.023 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:34 smithi016 ceph-mon[37215]: pgmap v1825: 176 pgs: 176 active+clean; 698 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:53:35.023 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:34 smithi016 ceph-mon[37215]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:53:35.023 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:34 smithi016 ceph-mon[37215]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T04:53:35.023 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:34 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3203726836' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "pool2": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T04:53:35.023 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:34 smithi016 ceph-mon[37215]: osdmap e822: 8 total, 8 up, 8 in 2024-03-26T04:53:35.023 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:34 smithi016 ceph-mon[37215]: from='client.? v1:172.21.15.16:0/3203726836' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "pool2": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:53:35.023 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:34 smithi016 systemd[1]: Stopping Ceph mon.a for d7d59826-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:53:35.023 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:34 smithi016 ceph-mon[29101]: pgmap v1825: 176 pgs: 176 active+clean; 698 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:53:35.023 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:34 smithi016 ceph-mon[29101]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:53:35.024 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:34 smithi016 ceph-mon[29101]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T04:53:35.024 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:34 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3203726836' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "pool2": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T04:53:35.024 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:34 smithi016 ceph-mon[29101]: osdmap e822: 8 total, 8 up, 8 in 2024-03-26T04:53:35.024 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:34 smithi016 ceph-mon[29101]: from='client.? v1:172.21.15.16:0/3203726836' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "pool2": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:53:35.037 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:34 smithi183 ceph-mon[33745]: pgmap v1825: 176 pgs: 176 active+clean; 698 KiB data, 896 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-26T04:53:35.037 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:34 smithi183 ceph-mon[33745]: from='client.14574 v1:172.21.15.183:0/3905977161' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-26T04:53:35.037 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:34 smithi183 ceph-mon[33745]: Health check cleared: POOL_FULL (was: 1 pool(s) full) 2024-03-26T04:53:35.037 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:34 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3203726836' entity='client.admin' cmd='[{"prefix": "osd pool delete", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "pool2": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "yes_i_really_really_mean_it": true}]': finished 2024-03-26T04:53:35.037 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:34 smithi183 ceph-mon[33745]: osdmap e822: 8 total, 8 up, 8 in 2024-03-26T04:53:35.037 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:34 smithi183 ceph-mon[33745]: from='client.? v1:172.21.15.16:0/3203726836' entity='client.admin' cmd=[{"prefix": "osd pool delete", "pool": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "pool2": "dcd349bf-3588-4b0c-9f5d-114ed7e44baf", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-26T04:53:35.323 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:35 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mon-a[29097]: 2024-03-26T04:53:35.026+0000 7f41f040e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:53:35.323 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:35 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mon-a[29097]: 2024-03-26T04:53:35.026+0000 7f41f040e700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-26T04:53:35.323 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:35 smithi016 podman[151933]: 2024-03-26 04:53:35.05762007 +0000 UTC m=+0.179897799 container died 4eb1141880f76e78f97440d7f4a860c981dc4f24726fca4ecbbccdaa2b3b8cea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mon-a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124) 2024-03-26T04:53:35.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:35 smithi016 podman[151933]: 2024-03-26 04:53:35.527477831 +0000 UTC m=+0.649755608 container cleanup 4eb1141880f76e78f97440d7f4a860c981dc4f24726fca4ecbbccdaa2b3b8cea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mon-a, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T04:53:35.658 INFO:journalctl@ceph.mon.a.smithi016.stdout:Mar 26 04:53:35 smithi016 bash[151933]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mon-a 2024-03-26T04:53:36.093 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d7d59826-eb26-11ee-95d0-87774f69a715@mon.a.service' 2024-03-26T04:53:36.132 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:53:36.132 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-26T04:53:36.132 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-03-26T04:53:36.132 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-d7d59826-eb26-11ee-95d0-87774f69a715@mon.c 2024-03-26T04:53:36.628 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:36 smithi016 systemd[1]: Stopping Ceph mon.c for d7d59826-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:53:36.628 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:36 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mon-c[37211]: 2024-03-26T04:53:36.447+0000 7f23062b1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:53:36.628 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:36 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mon-c[37211]: 2024-03-26T04:53:36.447+0000 7f23062b1700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-03-26T04:53:36.907 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:36 smithi016 podman[152056]: 2024-03-26 04:53:36.63279358 +0000 UTC m=+0.274803990 container died 2d20f53bd9c70a6c1116fe28f7ff96f3ff67dc8a3984598780567611f352f49c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mon-c, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319) 2024-03-26T04:53:37.366 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:36 smithi016 podman[152056]: 2024-03-26 04:53:36.956697769 +0000 UTC m=+0.598708162 container cleanup 2d20f53bd9c70a6c1116fe28f7ff96f3ff67dc8a3984598780567611f352f49c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, ceph=True) 2024-03-26T04:53:37.366 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:36 smithi016 bash[152056]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mon-c 2024-03-26T04:53:37.366 INFO:journalctl@ceph.mon.c.smithi016.stdout:Mar 26 04:53:37 smithi016 podman[152070]: 2024-03-26 04:53:37.174684317 +0000 UTC m=+0.537264329 container remove 2d20f53bd9c70a6c1116fe28f7ff96f3ff67dc8a3984598780567611f352f49c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mon-c, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-26T04:53:37.380 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d7d59826-eb26-11ee-95d0-87774f69a715@mon.c.service' 2024-03-26T04:53:37.417 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:53:37.417 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-03-26T04:53:37.417 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-03-26T04:53:37.417 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-d7d59826-eb26-11ee-95d0-87774f69a715@mon.b 2024-03-26T04:53:37.741 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:37 smithi183 systemd[1]: Stopping Ceph mon.b for d7d59826-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:53:38.005 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:37 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mon-b[33741]: 2024-03-26T04:53:37.744+0000 7f32c9e6d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:53:38.005 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:37 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mon-b[33741]: 2024-03-26T04:53:37.744+0000 7f32c9e6d700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-03-26T04:53:38.287 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:38 smithi183 podman[75781]: 2024-03-26 04:53:38.010085603 +0000 UTC m=+0.391371507 container died 2b01545a9662c3b2398e36cdb75d36131476fb1cbb0f2ce776865f5407309650 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mon-b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_CLEAN=True, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T04:53:38.683 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:38 smithi183 podman[75781]: 2024-03-26 04:53:38.520678882 +0000 UTC m=+0.901964798 container cleanup 2b01545a9662c3b2398e36cdb75d36131476fb1cbb0f2ce776865f5407309650 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mon-b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-5e70124, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T04:53:38.684 INFO:journalctl@ceph.mon.b.smithi183.stdout:Mar 26 04:53:38 smithi183 bash[75781]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mon-b 2024-03-26T04:53:38.803 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d7d59826-eb26-11ee-95d0-87774f69a715@mon.b.service' 2024-03-26T04:53:38.854 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:53:38.855 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-03-26T04:53:38.855 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-26T04:53:38.855 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-d7d59826-eb26-11ee-95d0-87774f69a715@mgr.y 2024-03-26T04:53:39.148 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:53:38 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: ::ffff:172.21.15.183 - - [26/Mar/2024:04:53:38] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-26T04:53:39.149 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:53:38 smithi016 systemd[1]: Stopping Ceph mgr.y for d7d59826-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:53:39.149 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:53:39 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:53:39.151+0000 7f92bb949700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:53:39.408 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:53:39 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y[29330]: 2024-03-26T04:53:39.151+0000 7f92bb949700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-26T04:53:39.408 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:53:39 smithi016 podman[152179]: 2024-03-26 04:53:39.184090928 +0000 UTC m=+0.131366162 container died 5d45f0a2de4bce6b21502d782b10c67257c33473df31d550c0faac81de87ad3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-26T04:53:39.825 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:53:39 smithi016 podman[152179]: 2024-03-26 04:53:39.603226398 +0000 UTC m=+0.550501609 container cleanup 5d45f0a2de4bce6b21502d782b10c67257c33473df31d550c0faac81de87ad3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-26T04:53:39.825 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:53:39 smithi016 bash[152179]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y 2024-03-26T04:53:40.068 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d7d59826-eb26-11ee-95d0-87774f69a715@mgr.y.service' 2024-03-26T04:53:40.092 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:53:39 smithi016 podman[152194]: 2024-03-26 04:53:39.829888783 +0000 UTC m=+0.645842898 container remove 5d45f0a2de4bce6b21502d782b10c67257c33473df31d550c0faac81de87ad3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5e70124b46c2a671efcaae4690cd9ea72670710a, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-y, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-26T04:53:40.093 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:53:40 smithi016 systemd[1]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715@mgr.y.service: Deactivated successfully. 2024-03-26T04:53:40.093 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:53:40 smithi016 systemd[1]: Stopped Ceph mgr.y for d7d59826-eb26-11ee-95d0-87774f69a715. 2024-03-26T04:53:40.093 INFO:journalctl@ceph.mgr.y.smithi016.stdout:Mar 26 04:53:40 smithi016 systemd[1]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715@mgr.y.service: Consumed 2min 10.744s CPU time. 2024-03-26T04:53:40.105 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:53:40.106 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-26T04:53:40.106 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-26T04:53:40.106 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-d7d59826-eb26-11ee-95d0-87774f69a715@mgr.x 2024-03-26T04:53:40.418 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:53:40 smithi183 systemd[1]: Stopping Ceph mgr.x for d7d59826-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:53:40.787 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:53:40 smithi183 podman[75903]: 2024-03-26 04:53:40.422814071 +0000 UTC m=+0.122596581 container died be7c6ed09b1cbebdd09c182e6fb34d547ebd336c2c3a700bc85052c8dff0c7be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-26T04:53:41.146 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:53:40 smithi183 podman[75903]: 2024-03-26 04:53:40.916544355 +0000 UTC m=+0.616326876 container cleanup be7c6ed09b1cbebdd09c182e6fb34d547ebd336c2c3a700bc85052c8dff0c7be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124) 2024-03-26T04:53:41.146 INFO:journalctl@ceph.mgr.x.smithi183.stdout:Mar 26 04:53:40 smithi183 bash[75903]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715-mgr-x 2024-03-26T04:53:41.279 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d7d59826-eb26-11ee-95d0-87774f69a715@mgr.x.service' 2024-03-26T04:53:41.316 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:53:41.316 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-26T04:53:41.316 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-26T04:53:41.316 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-d7d59826-eb26-11ee-95d0-87774f69a715@osd.0 2024-03-26T04:53:41.602 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 26 04:53:41 smithi016 systemd[1]: Stopping Ceph osd.0 for d7d59826-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:53:41.908 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 26 04:53:41 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-0[41205]: 2024-03-26T04:53:41.607+0000 7f08d1e90700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:53:41.908 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 26 04:53:41 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-0[41205]: 2024-03-26T04:53:41.607+0000 7f08d1e90700 -1 osd.0 822 *** Got signal Terminated *** 2024-03-26T04:53:41.908 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 26 04:53:41 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-0[41205]: 2024-03-26T04:53:41.607+0000 7f08d1e90700 -1 osd.0 822 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T04:53:47.270 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 26 04:53:46 smithi016 podman[152302]: 2024-03-26 04:53:46.94793252 +0000 UTC m=+5.436374078 container died abd9a0b5cc97847f30e0fcf24e1bc5a9431ed348b8aedd4c21d63cb58ed6e2d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS) 2024-03-26T04:53:47.562 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 26 04:53:47 smithi016 podman[152302]: 2024-03-26 04:53:47.365730853 +0000 UTC m=+5.854172409 container cleanup abd9a0b5cc97847f30e0fcf24e1bc5a9431ed348b8aedd4c21d63cb58ed6e2d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-26T04:53:47.562 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 26 04:53:47 smithi016 bash[152302]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-0 2024-03-26T04:53:47.896 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 26 04:53:47 smithi016 podman[152317]: 2024-03-26 04:53:47.567227657 +0000 UTC m=+0.614781835 container remove abd9a0b5cc97847f30e0fcf24e1bc5a9431ed348b8aedd4c21d63cb58ed6e2d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319) 2024-03-26T04:53:48.157 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 26 04:53:47 smithi016 podman[152410]: 2024-03-26 04:53:47.803937115 +0000 UTC m=+0.017104660 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T04:53:48.158 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 26 04:53:47 smithi016 podman[152410]: 2024-03-26 04:53:47.901743702 +0000 UTC m=+0.114911294 container create 5a637fdf6f23215c65a865ecd98407a23725f0cc3b9d1bbc66e4f93e9408f03e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124) 2024-03-26T04:53:48.158 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 26 04:53:48 smithi016 podman[152410]: 2024-03-26 04:53:48.137822317 +0000 UTC m=+0.350989859 container init 5a637fdf6f23215c65a865ecd98407a23725f0cc3b9d1bbc66e4f93e9408f03e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T04:53:48.158 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 26 04:53:48 smithi016 podman[152410]: 2024-03-26 04:53:48.141336422 +0000 UTC m=+0.354503956 container start 5a637fdf6f23215c65a865ecd98407a23725f0cc3b9d1bbc66e4f93e9408f03e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, org.label-schema.license=GPLv2) 2024-03-26T04:53:48.435 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 26 04:53:48 smithi016 podman[152410]: 2024-03-26 04:53:48.229193412 +0000 UTC m=+0.442360954 container attach 5a637fdf6f23215c65a865ecd98407a23725f0cc3b9d1bbc66e4f93e9408f03e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, io.buildah.version=1.33.5) 2024-03-26T04:53:48.853 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 26 04:53:48 smithi016 podman[152410]: 2024-03-26 04:53:48.440866314 +0000 UTC m=+0.654033848 container died 5a637fdf6f23215c65a865ecd98407a23725f0cc3b9d1bbc66e4f93e9408f03e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124) 2024-03-26T04:53:49.111 INFO:journalctl@ceph.osd.0.smithi016.stdout:Mar 26 04:53:49 smithi016 podman[152436]: 2024-03-26 04:53:49.099620146 +0000 UTC m=+0.653110184 container remove 5a637fdf6f23215c65a865ecd98407a23725f0cc3b9d1bbc66e4f93e9408f03e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-26T04:53:49.116 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d7d59826-eb26-11ee-95d0-87774f69a715@osd.0.service' 2024-03-26T04:53:49.160 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:53:49.161 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-26T04:53:49.161 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-26T04:53:49.161 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-d7d59826-eb26-11ee-95d0-87774f69a715@osd.1 2024-03-26T04:53:49.391 INFO:journalctl@ceph.osd.1.smithi016.stdout:Mar 26 04:53:49 smithi016 systemd[1]: Stopping Ceph osd.1 for d7d59826-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:53:49.657 INFO:journalctl@ceph.osd.1.smithi016.stdout:Mar 26 04:53:49 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-1[46918]: 2024-03-26T04:53:49.510+0000 7ff494738700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:53:49.658 INFO:journalctl@ceph.osd.1.smithi016.stdout:Mar 26 04:53:49 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-1[46918]: 2024-03-26T04:53:49.510+0000 7ff494738700 -1 osd.1 822 *** Got signal Terminated *** 2024-03-26T04:53:49.658 INFO:journalctl@ceph.osd.1.smithi016.stdout:Mar 26 04:53:49 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-1[46918]: 2024-03-26T04:53:49.510+0000 7ff494738700 -1 osd.1 822 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T04:53:55.347 INFO:journalctl@ceph.osd.1.smithi016.stdout:Mar 26 04:53:55 smithi016 podman[152519]: 2024-03-26 04:53:55.013571837 +0000 UTC m=+5.617287179 container died 76e730fce3704db3455b2f9cafcee744a8a4b19ab7f700631ab3cbbe8d2c9ad0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-1, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-26T04:53:55.348 INFO:journalctl@ceph.osd.1.smithi016.stdout:Mar 26 04:53:55 smithi016 podman[152519]: 2024-03-26 04:53:55.352555075 +0000 UTC m=+5.956270400 container cleanup 76e730fce3704db3455b2f9cafcee744a8a4b19ab7f700631ab3cbbe8d2c9ad0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-1, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-26T04:53:55.657 INFO:journalctl@ceph.osd.1.smithi016.stdout:Mar 26 04:53:55 smithi016 bash[152519]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-1 2024-03-26T04:53:55.657 INFO:journalctl@ceph.osd.1.smithi016.stdout:Mar 26 04:53:55 smithi016 podman[152546]: 2024-03-26 04:53:55.562495154 +0000 UTC m=+0.551442411 container remove 76e730fce3704db3455b2f9cafcee744a8a4b19ab7f700631ab3cbbe8d2c9ad0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-1, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124) 2024-03-26T04:53:56.229 INFO:journalctl@ceph.osd.1.smithi016.stdout:Mar 26 04:53:55 smithi016 podman[152642]: 2024-03-26 04:53:55.878487158 +0000 UTC m=+0.018143577 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T04:53:56.229 INFO:journalctl@ceph.osd.1.smithi016.stdout:Mar 26 04:53:56 smithi016 podman[152642]: 2024-03-26 04:53:56.10499829 +0000 UTC m=+0.244654707 container create 232516d9dde686c01f3b9d48ec8007731946de94c571b0ff7b9b976d6707e26d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-1-deactivate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-26T04:53:56.623 INFO:journalctl@ceph.osd.1.smithi016.stdout:Mar 26 04:53:56 smithi016 podman[152642]: 2024-03-26 04:53:56.324060263 +0000 UTC m=+0.463716680 container init 232516d9dde686c01f3b9d48ec8007731946de94c571b0ff7b9b976d6707e26d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-26T04:53:56.623 INFO:journalctl@ceph.osd.1.smithi016.stdout:Mar 26 04:53:56 smithi016 podman[152642]: 2024-03-26 04:53:56.327033841 +0000 UTC m=+0.466690258 container start 232516d9dde686c01f3b9d48ec8007731946de94c571b0ff7b9b976d6707e26d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-26T04:53:56.623 INFO:journalctl@ceph.osd.1.smithi016.stdout:Mar 26 04:53:56 smithi016 podman[152642]: 2024-03-26 04:53:56.398870949 +0000 UTC m=+0.538527370 container attach 232516d9dde686c01f3b9d48ec8007731946de94c571b0ff7b9b976d6707e26d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-26T04:53:56.907 INFO:journalctl@ceph.osd.1.smithi016.stdout:Mar 26 04:53:56 smithi016 podman[152642]: 2024-03-26 04:53:56.628444916 +0000 UTC m=+0.768101333 container died 232516d9dde686c01f3b9d48ec8007731946de94c571b0ff7b9b976d6707e26d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-5e70124) 2024-03-26T04:53:57.226 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d7d59826-eb26-11ee-95d0-87774f69a715@osd.1.service' 2024-03-26T04:53:57.268 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:53:57.268 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-26T04:53:57.268 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-26T04:53:57.269 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-d7d59826-eb26-11ee-95d0-87774f69a715@osd.2 2024-03-26T04:53:57.585 INFO:journalctl@ceph.osd.2.smithi016.stdout:Mar 26 04:53:57 smithi016 systemd[1]: Stopping Ceph osd.2 for d7d59826-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:53:57.907 INFO:journalctl@ceph.osd.2.smithi016.stdout:Mar 26 04:53:57 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-2[52956]: 2024-03-26T04:53:57.588+0000 7f0e8204a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:53:57.907 INFO:journalctl@ceph.osd.2.smithi016.stdout:Mar 26 04:53:57 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-2[52956]: 2024-03-26T04:53:57.588+0000 7f0e8204a700 -1 osd.2 822 *** Got signal Terminated *** 2024-03-26T04:53:57.908 INFO:journalctl@ceph.osd.2.smithi016.stdout:Mar 26 04:53:57 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-2[52956]: 2024-03-26T04:53:57.588+0000 7f0e8204a700 -1 osd.2 822 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T04:54:03.184 INFO:journalctl@ceph.osd.2.smithi016.stdout:Mar 26 04:54:02 smithi016 podman[152749]: 2024-03-26 04:54:02.916800121 +0000 UTC m=+5.423882803 container died 808f593152e6c614c688839229181b3d54003f2145fbb1c95b8ae7d09860b4f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=squid-5e70124) 2024-03-26T04:54:03.560 INFO:journalctl@ceph.osd.2.smithi016.stdout:Mar 26 04:54:03 smithi016 podman[152749]: 2024-03-26 04:54:03.280390162 +0000 UTC m=+5.787472841 container cleanup 808f593152e6c614c688839229181b3d54003f2145fbb1c95b8ae7d09860b4f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-26T04:54:03.560 INFO:journalctl@ceph.osd.2.smithi016.stdout:Mar 26 04:54:03 smithi016 bash[152749]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-2 2024-03-26T04:54:03.893 INFO:journalctl@ceph.osd.2.smithi016.stdout:Mar 26 04:54:03 smithi016 podman[152769]: 2024-03-26 04:54:03.565102916 +0000 UTC m=+0.649201416 container remove 808f593152e6c614c688839229181b3d54003f2145fbb1c95b8ae7d09860b4f4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-26T04:54:04.157 INFO:journalctl@ceph.osd.2.smithi016.stdout:Mar 26 04:54:03 smithi016 podman[152860]: 2024-03-26 04:54:03.802125196 +0000 UTC m=+0.017506135 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T04:54:04.157 INFO:journalctl@ceph.osd.2.smithi016.stdout:Mar 26 04:54:03 smithi016 podman[152860]: 2024-03-26 04:54:03.907798578 +0000 UTC m=+0.123179508 container create 9635ab224347c388f16df93c948a54c20806db3c50d84e4fc304109a4ab116cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, RELEASE=squid-5e70124) 2024-03-26T04:54:04.158 INFO:journalctl@ceph.osd.2.smithi016.stdout:Mar 26 04:54:04 smithi016 podman[152860]: 2024-03-26 04:54:04.118588713 +0000 UTC m=+0.333969645 container init 9635ab224347c388f16df93c948a54c20806db3c50d84e4fc304109a4ab116cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-26T04:54:04.158 INFO:journalctl@ceph.osd.2.smithi016.stdout:Mar 26 04:54:04 smithi016 podman[152860]: 2024-03-26 04:54:04.121517996 +0000 UTC m=+0.336898926 container start 9635ab224347c388f16df93c948a54c20806db3c50d84e4fc304109a4ab116cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-26T04:54:04.657 INFO:journalctl@ceph.osd.2.smithi016.stdout:Mar 26 04:54:04 smithi016 podman[152860]: 2024-03-26 04:54:04.193505873 +0000 UTC m=+0.408886818 container attach 9635ab224347c388f16df93c948a54c20806db3c50d84e4fc304109a4ab116cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T04:54:04.658 INFO:journalctl@ceph.osd.2.smithi016.stdout:Mar 26 04:54:04 smithi016 podman[152860]: 2024-03-26 04:54:04.41803877 +0000 UTC m=+0.633419702 container died 9635ab224347c388f16df93c948a54c20806db3c50d84e4fc304109a4ab116cb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T04:54:04.980 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d7d59826-eb26-11ee-95d0-87774f69a715@osd.2.service' 2024-03-26T04:54:05.022 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:54:05.023 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-26T04:54:05.023 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-26T04:54:05.023 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-d7d59826-eb26-11ee-95d0-87774f69a715@osd.3 2024-03-26T04:54:05.345 INFO:journalctl@ceph.osd.3.smithi016.stdout:Mar 26 04:54:05 smithi016 systemd[1]: Stopping Ceph osd.3 for d7d59826-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:54:05.657 INFO:journalctl@ceph.osd.3.smithi016.stdout:Mar 26 04:54:05 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-3[58771]: 2024-03-26T04:54:05.349+0000 7f83a0a62700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:54:05.657 INFO:journalctl@ceph.osd.3.smithi016.stdout:Mar 26 04:54:05 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-3[58771]: 2024-03-26T04:54:05.349+0000 7f83a0a62700 -1 osd.3 822 *** Got signal Terminated *** 2024-03-26T04:54:05.657 INFO:journalctl@ceph.osd.3.smithi016.stdout:Mar 26 04:54:05 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-3[58771]: 2024-03-26T04:54:05.349+0000 7f83a0a62700 -1 osd.3 822 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T04:54:08.537 INFO:journalctl@ceph.osd.4.smithi183.stdout:Mar 26 04:54:08 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-4[38631]: 2024-03-26T04:54:08.211+0000 7fada2ab5700 -1 osd.4 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:42.291707+0000 front 2024-03-26T04:53:42.291544+0000 (oldest deadline 2024-03-26T04:54:08.191581+0000) 2024-03-26T04:54:09.536 INFO:journalctl@ceph.osd.4.smithi183.stdout:Mar 26 04:54:09 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-4[38631]: 2024-03-26T04:54:09.174+0000 7fada2ab5700 -1 osd.4 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:42.291707+0000 front 2024-03-26T04:53:42.291544+0000 (oldest deadline 2024-03-26T04:54:08.191581+0000) 2024-03-26T04:54:10.037 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 26 04:54:09 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-5[44571]: 2024-03-26T04:54:09.757+0000 7fd005367700 -1 osd.5 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.276691+0000 front 2024-03-26T04:53:46.276765+0000 (oldest deadline 2024-03-26T04:54:09.677086+0000) 2024-03-26T04:54:10.037 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:09 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:09.668+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.143968+0000 front 2024-03-26T04:53:46.143727+0000 (oldest deadline 2024-03-26T04:54:09.043713+0000) 2024-03-26T04:54:10.536 INFO:journalctl@ceph.osd.4.smithi183.stdout:Mar 26 04:54:10 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-4[38631]: 2024-03-26T04:54:10.201+0000 7fada2ab5700 -1 osd.4 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:42.291707+0000 front 2024-03-26T04:53:42.291544+0000 (oldest deadline 2024-03-26T04:54:08.191581+0000) 2024-03-26T04:54:10.537 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:10 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:10.400+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:44.196054+0000 front 2024-03-26T04:53:44.196039+0000 (oldest deadline 2024-03-26T04:54:10.095928+0000) 2024-03-26T04:54:11.037 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 26 04:54:10 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-5[44571]: 2024-03-26T04:54:10.761+0000 7fd005367700 -1 osd.5 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.276691+0000 front 2024-03-26T04:53:46.276765+0000 (oldest deadline 2024-03-26T04:54:09.677086+0000) 2024-03-26T04:54:11.037 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:10 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:10.662+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.143968+0000 front 2024-03-26T04:53:46.143727+0000 (oldest deadline 2024-03-26T04:54:09.043713+0000) 2024-03-26T04:54:11.128 INFO:journalctl@ceph.osd.3.smithi016.stdout:Mar 26 04:54:10 smithi016 podman[152964]: 2024-03-26 04:54:10.825900119 +0000 UTC m=+5.569716541 container died f5c56a581326e6c9ef73a5d8bb0d3dfe9fc41f20d43011e125f56de27868b729 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-3, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True) 2024-03-26T04:54:11.407 INFO:journalctl@ceph.osd.3.smithi016.stdout:Mar 26 04:54:11 smithi016 podman[152964]: 2024-03-26 04:54:11.23328223 +0000 UTC m=+5.977098664 container cleanup f5c56a581326e6c9ef73a5d8bb0d3dfe9fc41f20d43011e125f56de27868b729 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-3, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, RELEASE=squid-5e70124) 2024-03-26T04:54:11.407 INFO:journalctl@ceph.osd.3.smithi016.stdout:Mar 26 04:54:11 smithi016 bash[152964]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-3 2024-03-26T04:54:11.536 INFO:journalctl@ceph.osd.4.smithi183.stdout:Mar 26 04:54:11 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-4[38631]: 2024-03-26T04:54:11.217+0000 7fada2ab5700 -1 osd.4 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:42.291707+0000 front 2024-03-26T04:53:42.291544+0000 (oldest deadline 2024-03-26T04:54:08.191581+0000) 2024-03-26T04:54:11.537 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:11 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:11.396+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:44.196054+0000 front 2024-03-26T04:53:44.196039+0000 (oldest deadline 2024-03-26T04:54:10.095928+0000) 2024-03-26T04:54:11.795 INFO:journalctl@ceph.osd.3.smithi016.stdout:Mar 26 04:54:11 smithi016 podman[152984]: 2024-03-26 04:54:11.467866585 +0000 UTC m=+0.634148222 container remove f5c56a581326e6c9ef73a5d8bb0d3dfe9fc41f20d43011e125f56de27868b729 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-26T04:54:12.037 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 26 04:54:11 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-5[44571]: 2024-03-26T04:54:11.781+0000 7fd005367700 -1 osd.5 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.276691+0000 front 2024-03-26T04:53:46.276765+0000 (oldest deadline 2024-03-26T04:54:09.677086+0000) 2024-03-26T04:54:12.037 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:11 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:11.668+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.143968+0000 front 2024-03-26T04:53:46.143727+0000 (oldest deadline 2024-03-26T04:54:09.043713+0000) 2024-03-26T04:54:12.116 INFO:journalctl@ceph.osd.3.smithi016.stdout:Mar 26 04:54:11 smithi016 podman[153078]: 2024-03-26 04:54:11.703769808 +0000 UTC m=+0.017823791 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T04:54:12.116 INFO:journalctl@ceph.osd.3.smithi016.stdout:Mar 26 04:54:11 smithi016 podman[153078]: 2024-03-26 04:54:11.827058971 +0000 UTC m=+0.141112954 container create 8167a0baca3aa76b2cc02a3ce88662efe5f4b781fcadf123479a3a21d9494a73 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-3-deactivate, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319) 2024-03-26T04:54:12.116 INFO:journalctl@ceph.osd.3.smithi016.stdout:Mar 26 04:54:12 smithi016 podman[153078]: 2024-03-26 04:54:12.046059849 +0000 UTC m=+0.360113833 container init 8167a0baca3aa76b2cc02a3ce88662efe5f4b781fcadf123479a3a21d9494a73 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T04:54:12.116 INFO:journalctl@ceph.osd.3.smithi016.stdout:Mar 26 04:54:12 smithi016 podman[153078]: 2024-03-26 04:54:12.049072666 +0000 UTC m=+0.363126648 container start 8167a0baca3aa76b2cc02a3ce88662efe5f4b781fcadf123479a3a21d9494a73 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T04:54:12.116 INFO:journalctl@ceph.osd.3.smithi016.stdout:Mar 26 04:54:12 smithi016 podman[153078]: 2024-03-26 04:54:12.120899245 +0000 UTC m=+0.434953231 container attach 8167a0baca3aa76b2cc02a3ce88662efe5f4b781fcadf123479a3a21d9494a73 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-3-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-26T04:54:12.407 INFO:journalctl@ceph.osd.3.smithi016.stdout:Mar 26 04:54:12 smithi016 podman[153078]: 2024-03-26 04:54:12.339711111 +0000 UTC m=+0.653765098 container died 8167a0baca3aa76b2cc02a3ce88662efe5f4b781fcadf123479a3a21d9494a73 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-3-deactivate, ceph=True, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T04:54:12.537 INFO:journalctl@ceph.osd.4.smithi183.stdout:Mar 26 04:54:12 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-4[38631]: 2024-03-26T04:54:12.193+0000 7fada2ab5700 -1 osd.4 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:42.291707+0000 front 2024-03-26T04:53:42.291544+0000 (oldest deadline 2024-03-26T04:54:08.191581+0000) 2024-03-26T04:54:12.537 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:12 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:12.416+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:44.196054+0000 front 2024-03-26T04:53:44.196039+0000 (oldest deadline 2024-03-26T04:54:10.095928+0000) 2024-03-26T04:54:12.877 INFO:journalctl@ceph.osd.3.smithi016.stdout:Mar 26 04:54:12 smithi016 podman[153104]: 2024-03-26 04:54:12.866046004 +0000 UTC m=+0.521244580 container remove 8167a0baca3aa76b2cc02a3ce88662efe5f4b781fcadf123479a3a21d9494a73 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-26T04:54:12.883 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d7d59826-eb26-11ee-95d0-87774f69a715@osd.3.service' 2024-03-26T04:54:12.925 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:54:12.925 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-26T04:54:12.925 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-26T04:54:12.926 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-d7d59826-eb26-11ee-95d0-87774f69a715@osd.4 2024-03-26T04:54:12.957 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 26 04:54:12 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-5[44571]: 2024-03-26T04:54:12.747+0000 7fd005367700 -1 osd.5 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.276691+0000 front 2024-03-26T04:53:46.276765+0000 (oldest deadline 2024-03-26T04:54:09.677086+0000) 2024-03-26T04:54:12.957 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:12 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:12.691+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.143968+0000 front 2024-03-26T04:53:46.143727+0000 (oldest deadline 2024-03-26T04:54:09.043713+0000) 2024-03-26T04:54:13.238 INFO:journalctl@ceph.osd.4.smithi183.stdout:Mar 26 04:54:12 smithi183 systemd[1]: Stopping Ceph osd.4 for d7d59826-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:54:13.238 INFO:journalctl@ceph.osd.4.smithi183.stdout:Mar 26 04:54:13 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-4[38631]: 2024-03-26T04:54:13.194+0000 7fada2ab5700 -1 osd.4 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:42.291707+0000 front 2024-03-26T04:53:42.291544+0000 (oldest deadline 2024-03-26T04:54:08.191581+0000) 2024-03-26T04:54:13.238 INFO:journalctl@ceph.osd.4.smithi183.stdout:Mar 26 04:54:13 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-4[38631]: 2024-03-26T04:54:13.242+0000 7fadaa4c9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:54:13.238 INFO:journalctl@ceph.osd.4.smithi183.stdout:Mar 26 04:54:13 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-4[38631]: 2024-03-26T04:54:13.242+0000 7fadaa4c9700 -1 osd.4 822 *** Got signal Terminated *** 2024-03-26T04:54:13.238 INFO:journalctl@ceph.osd.4.smithi183.stdout:Mar 26 04:54:13 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-4[38631]: 2024-03-26T04:54:13.242+0000 7fadaa4c9700 -1 osd.4 822 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T04:54:13.536 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:13 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:13.458+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:44.196054+0000 front 2024-03-26T04:53:44.196039+0000 (oldest deadline 2024-03-26T04:54:10.095928+0000) 2024-03-26T04:54:14.037 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 26 04:54:13 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-5[44571]: 2024-03-26T04:54:13.702+0000 7fd005367700 -1 osd.5 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.276691+0000 front 2024-03-26T04:53:46.276765+0000 (oldest deadline 2024-03-26T04:54:09.677086+0000) 2024-03-26T04:54:14.037 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:13 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:13.696+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.143968+0000 front 2024-03-26T04:53:46.143727+0000 (oldest deadline 2024-03-26T04:54:09.043713+0000) 2024-03-26T04:54:14.496 INFO:journalctl@ceph.osd.4.smithi183.stdout:Mar 26 04:54:14 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-4[38631]: 2024-03-26T04:54:14.173+0000 7fada2ab5700 -1 osd.4 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:42.291707+0000 front 2024-03-26T04:53:42.291544+0000 (oldest deadline 2024-03-26T04:54:08.191581+0000) 2024-03-26T04:54:14.496 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:54:14 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[65252]: ts=2024-03-26T04:54:14.150Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.16:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.16:8765: connect: connection refused" 2024-03-26T04:54:14.496 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:54:14 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[65252]: ts=2024-03-26T04:54:14.151Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.16:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.16:8765: connect: connection refused" 2024-03-26T04:54:14.496 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:54:14 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[65252]: ts=2024-03-26T04:54:14.153Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.16:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.16:8765: connect: connection refused" 2024-03-26T04:54:14.496 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:54:14 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[65252]: ts=2024-03-26T04:54:14.153Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.16:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.16:8765: connect: connection refused" 2024-03-26T04:54:14.787 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:14 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:14.500+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:44.196054+0000 front 2024-03-26T04:53:44.196039+0000 (oldest deadline 2024-03-26T04:54:10.095928+0000) 2024-03-26T04:54:14.787 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 26 04:54:14 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-5[44571]: 2024-03-26T04:54:14.697+0000 7fd005367700 -1 osd.5 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.276691+0000 front 2024-03-26T04:53:46.276765+0000 (oldest deadline 2024-03-26T04:54:09.677086+0000) 2024-03-26T04:54:14.787 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:14 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:14.686+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.143968+0000 front 2024-03-26T04:53:46.143727+0000 (oldest deadline 2024-03-26T04:54:09.043713+0000) 2024-03-26T04:54:15.510 INFO:journalctl@ceph.osd.4.smithi183.stdout:Mar 26 04:54:15 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-4[38631]: 2024-03-26T04:54:15.221+0000 7fada2ab5700 -1 osd.4 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:42.291707+0000 front 2024-03-26T04:53:42.291544+0000 (oldest deadline 2024-03-26T04:54:08.191581+0000) 2024-03-26T04:54:15.786 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:15 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:15.514+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:44.196054+0000 front 2024-03-26T04:53:44.196039+0000 (oldest deadline 2024-03-26T04:54:10.095928+0000) 2024-03-26T04:54:15.787 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 26 04:54:15 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-5[44571]: 2024-03-26T04:54:15.747+0000 7fd005367700 -1 osd.5 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.276691+0000 front 2024-03-26T04:53:46.276765+0000 (oldest deadline 2024-03-26T04:54:09.677086+0000) 2024-03-26T04:54:15.787 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:15 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:15.673+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.143968+0000 front 2024-03-26T04:53:46.143727+0000 (oldest deadline 2024-03-26T04:54:09.043713+0000) 2024-03-26T04:54:15.787 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:15 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:15.673+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:54.245099+0000 front 2024-03-26T04:53:54.245259+0000 (oldest deadline 2024-03-26T04:54:15.345177+0000) 2024-03-26T04:54:16.519 INFO:journalctl@ceph.osd.4.smithi183.stdout:Mar 26 04:54:16 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-4[38631]: 2024-03-26T04:54:16.200+0000 7fada2ab5700 -1 osd.4 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:42.291707+0000 front 2024-03-26T04:53:42.291544+0000 (oldest deadline 2024-03-26T04:54:08.191581+0000) 2024-03-26T04:54:16.786 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 26 04:54:16 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-5[44571]: 2024-03-26T04:54:16.742+0000 7fd005367700 -1 osd.5 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.276691+0000 front 2024-03-26T04:53:46.276765+0000 (oldest deadline 2024-03-26T04:54:09.677086+0000) 2024-03-26T04:54:16.787 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:16 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:16.655+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.143968+0000 front 2024-03-26T04:53:46.143727+0000 (oldest deadline 2024-03-26T04:54:09.043713+0000) 2024-03-26T04:54:16.787 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:16 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:16.655+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:54.245099+0000 front 2024-03-26T04:53:54.245259+0000 (oldest deadline 2024-03-26T04:54:15.345177+0000) 2024-03-26T04:54:16.787 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:16 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:16.523+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:44.196054+0000 front 2024-03-26T04:53:44.196039+0000 (oldest deadline 2024-03-26T04:54:10.095928+0000) 2024-03-26T04:54:17.499 INFO:journalctl@ceph.osd.4.smithi183.stdout:Mar 26 04:54:17 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-4[38631]: 2024-03-26T04:54:17.233+0000 7fada2ab5700 -1 osd.4 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:42.291707+0000 front 2024-03-26T04:53:42.291544+0000 (oldest deadline 2024-03-26T04:54:08.191581+0000) 2024-03-26T04:54:17.775 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:17 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:17.503+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:44.196054+0000 front 2024-03-26T04:53:44.196039+0000 (oldest deadline 2024-03-26T04:54:10.095928+0000) 2024-03-26T04:54:17.775 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:17 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:17.646+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.143968+0000 front 2024-03-26T04:53:46.143727+0000 (oldest deadline 2024-03-26T04:54:09.043713+0000) 2024-03-26T04:54:17.775 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:17 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:17.646+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:54.245099+0000 front 2024-03-26T04:53:54.245259+0000 (oldest deadline 2024-03-26T04:54:15.345177+0000) 2024-03-26T04:54:18.036 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 26 04:54:17 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-5[44571]: 2024-03-26T04:54:17.779+0000 7fd005367700 -1 osd.5 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.276691+0000 front 2024-03-26T04:53:46.276765+0000 (oldest deadline 2024-03-26T04:54:09.677086+0000) 2024-03-26T04:54:18.520 INFO:journalctl@ceph.osd.4.smithi183.stdout:Mar 26 04:54:18 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-4[38631]: 2024-03-26T04:54:18.232+0000 7fada2ab5700 -1 osd.4 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:42.291707+0000 front 2024-03-26T04:53:42.291544+0000 (oldest deadline 2024-03-26T04:54:08.191581+0000) 2024-03-26T04:54:18.773 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:18 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:18.603+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.143968+0000 front 2024-03-26T04:53:46.143727+0000 (oldest deadline 2024-03-26T04:54:09.043713+0000) 2024-03-26T04:54:18.773 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:18 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:18.603+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:54.245099+0000 front 2024-03-26T04:53:54.245259+0000 (oldest deadline 2024-03-26T04:54:15.345177+0000) 2024-03-26T04:54:18.774 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:18 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:18.524+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:44.196054+0000 front 2024-03-26T04:53:44.196039+0000 (oldest deadline 2024-03-26T04:54:10.095928+0000) 2024-03-26T04:54:18.774 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:18 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:18.524+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:53.597025+0000 front 2024-03-26T04:53:53.597059+0000 (oldest deadline 2024-03-26T04:54:18.197443+0000) 2024-03-26T04:54:19.036 INFO:journalctl@ceph.osd.4.smithi183.stdout:Mar 26 04:54:18 smithi183 podman[76039]: 2024-03-26 04:54:18.778560667 +0000 UTC m=+5.645377424 container died e8353b297aedc59efaa4fc34597ba4dbbc9be5804b1fd1f9f7d3cb2e3b6dd4fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-26T04:54:19.037 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 26 04:54:18 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-5[44571]: 2024-03-26T04:54:18.818+0000 7fd005367700 -1 osd.5 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.276691+0000 front 2024-03-26T04:53:46.276765+0000 (oldest deadline 2024-03-26T04:54:09.677086+0000) 2024-03-26T04:54:19.037 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 26 04:54:18 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-5[44571]: 2024-03-26T04:54:18.818+0000 7fd005367700 -1 osd.5 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:53.678558+0000 front 2024-03-26T04:53:53.678535+0000 (oldest deadline 2024-03-26T04:54:18.279041+0000) 2024-03-26T04:54:19.414 INFO:journalctl@ceph.osd.4.smithi183.stdout:Mar 26 04:54:19 smithi183 podman[76039]: 2024-03-26 04:54:19.184306869 +0000 UTC m=+6.051123632 container cleanup e8353b297aedc59efaa4fc34597ba4dbbc9be5804b1fd1f9f7d3cb2e3b6dd4fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux ) 2024-03-26T04:54:19.414 INFO:journalctl@ceph.osd.4.smithi183.stdout:Mar 26 04:54:19 smithi183 bash[76039]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-4 2024-03-26T04:54:19.786 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:19 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:19.629+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.143968+0000 front 2024-03-26T04:53:46.143727+0000 (oldest deadline 2024-03-26T04:54:09.043713+0000) 2024-03-26T04:54:19.787 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:19 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:19.629+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:54.245099+0000 front 2024-03-26T04:53:54.245259+0000 (oldest deadline 2024-03-26T04:54:15.345177+0000) 2024-03-26T04:54:19.787 INFO:journalctl@ceph.osd.4.smithi183.stdout:Mar 26 04:54:19 smithi183 podman[76056]: 2024-03-26 04:54:19.419083908 +0000 UTC m=+0.641511263 container remove e8353b297aedc59efaa4fc34597ba4dbbc9be5804b1fd1f9f7d3cb2e3b6dd4fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-4, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, RELEASE=squid-5e70124) 2024-03-26T04:54:19.787 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:19 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:19.523+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:44.196054+0000 front 2024-03-26T04:53:44.196039+0000 (oldest deadline 2024-03-26T04:54:10.095928+0000) 2024-03-26T04:54:19.787 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:19 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:19.523+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:53.597025+0000 front 2024-03-26T04:53:53.597059+0000 (oldest deadline 2024-03-26T04:54:18.197443+0000) 2024-03-26T04:54:20.076 INFO:journalctl@ceph.osd.4.smithi183.stdout:Mar 26 04:54:19 smithi183 podman[76147]: 2024-03-26 04:54:19.758868389 +0000 UTC m=+0.021089586 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T04:54:20.076 INFO:journalctl@ceph.osd.4.smithi183.stdout:Mar 26 04:54:19 smithi183 podman[76147]: 2024-03-26 04:54:19.878610538 +0000 UTC m=+0.140831712 container create fede3728f8cee38e407e8e9fd1f2484cb7a5e0f638e748c3c5678388c427861f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-26T04:54:20.076 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 26 04:54:19 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-5[44571]: 2024-03-26T04:54:19.846+0000 7fd005367700 -1 osd.5 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.276691+0000 front 2024-03-26T04:53:46.276765+0000 (oldest deadline 2024-03-26T04:54:09.677086+0000) 2024-03-26T04:54:20.077 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 26 04:54:19 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-5[44571]: 2024-03-26T04:54:19.846+0000 7fd005367700 -1 osd.5 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:53.678558+0000 front 2024-03-26T04:53:53.678535+0000 (oldest deadline 2024-03-26T04:54:18.279041+0000) 2024-03-26T04:54:20.350 INFO:journalctl@ceph.osd.4.smithi183.stdout:Mar 26 04:54:20 smithi183 podman[76147]: 2024-03-26 04:54:20.081569832 +0000 UTC m=+0.343791018 container init fede3728f8cee38e407e8e9fd1f2484cb7a5e0f638e748c3c5678388c427861f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-4-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-26T04:54:20.351 INFO:journalctl@ceph.osd.4.smithi183.stdout:Mar 26 04:54:20 smithi183 podman[76147]: 2024-03-26 04:54:20.085190578 +0000 UTC m=+0.347411758 container start fede3728f8cee38e407e8e9fd1f2484cb7a5e0f638e748c3c5678388c427861f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS) 2024-03-26T04:54:20.351 INFO:journalctl@ceph.osd.4.smithi183.stdout:Mar 26 04:54:20 smithi183 podman[76147]: 2024-03-26 04:54:20.181332925 +0000 UTC m=+0.443554107 container attach fede3728f8cee38e407e8e9fd1f2484cb7a5e0f638e748c3c5678388c427861f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-26T04:54:20.705 INFO:journalctl@ceph.osd.4.smithi183.stdout:Mar 26 04:54:20 smithi183 podman[76147]: 2024-03-26 04:54:20.391318876 +0000 UTC m=+0.653540059 container died fede3728f8cee38e407e8e9fd1f2484cb7a5e0f638e748c3c5678388c427861f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-4-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T04:54:20.706 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:20 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:20.591+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.143968+0000 front 2024-03-26T04:53:46.143727+0000 (oldest deadline 2024-03-26T04:54:09.043713+0000) 2024-03-26T04:54:20.706 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:20 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:20.591+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:54.245099+0000 front 2024-03-26T04:53:54.245259+0000 (oldest deadline 2024-03-26T04:54:15.345177+0000) 2024-03-26T04:54:20.706 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:20 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:20.477+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:44.196054+0000 front 2024-03-26T04:53:44.196039+0000 (oldest deadline 2024-03-26T04:54:10.095928+0000) 2024-03-26T04:54:20.706 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:20 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:20.477+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:53.597025+0000 front 2024-03-26T04:53:53.597059+0000 (oldest deadline 2024-03-26T04:54:18.197443+0000) 2024-03-26T04:54:20.971 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 26 04:54:20 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-5[44571]: 2024-03-26T04:54:20.840+0000 7fd005367700 -1 osd.5 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.276691+0000 front 2024-03-26T04:53:46.276765+0000 (oldest deadline 2024-03-26T04:54:09.677086+0000) 2024-03-26T04:54:20.971 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 26 04:54:20 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-5[44571]: 2024-03-26T04:54:20.840+0000 7fd005367700 -1 osd.5 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:53.678558+0000 front 2024-03-26T04:53:53.678535+0000 (oldest deadline 2024-03-26T04:54:18.279041+0000) 2024-03-26T04:54:20.988 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d7d59826-eb26-11ee-95d0-87774f69a715@osd.4.service' 2024-03-26T04:54:21.068 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:54:21.068 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-26T04:54:21.068 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-26T04:54:21.068 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-d7d59826-eb26-11ee-95d0-87774f69a715@osd.5 2024-03-26T04:54:21.286 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 26 04:54:21 smithi183 systemd[1]: Stopping Ceph osd.5 for d7d59826-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:54:21.632 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 26 04:54:21 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-5[44571]: 2024-03-26T04:54:21.446+0000 7fd008d73700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:54:21.632 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 26 04:54:21 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-5[44571]: 2024-03-26T04:54:21.446+0000 7fd008d73700 -1 osd.5 822 *** Got signal Terminated *** 2024-03-26T04:54:21.632 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 26 04:54:21 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-5[44571]: 2024-03-26T04:54:21.446+0000 7fd008d73700 -1 osd.5 822 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T04:54:21.632 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:21 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:21.498+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:44.196054+0000 front 2024-03-26T04:53:44.196039+0000 (oldest deadline 2024-03-26T04:54:10.095928+0000) 2024-03-26T04:54:21.632 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:21 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:21.498+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:53.597025+0000 front 2024-03-26T04:53:53.597059+0000 (oldest deadline 2024-03-26T04:54:18.197443+0000) 2024-03-26T04:54:22.036 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 26 04:54:21 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-5[44571]: 2024-03-26T04:54:21.805+0000 7fd005367700 -1 osd.5 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.276691+0000 front 2024-03-26T04:53:46.276765+0000 (oldest deadline 2024-03-26T04:54:09.677086+0000) 2024-03-26T04:54:22.037 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 26 04:54:21 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-5[44571]: 2024-03-26T04:54:21.805+0000 7fd005367700 -1 osd.5 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:53.678558+0000 front 2024-03-26T04:53:53.678535+0000 (oldest deadline 2024-03-26T04:54:18.279041+0000) 2024-03-26T04:54:22.037 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:21 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:21.636+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.143968+0000 front 2024-03-26T04:53:46.143727+0000 (oldest deadline 2024-03-26T04:54:09.043713+0000) 2024-03-26T04:54:22.037 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:21 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:21.636+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:54.245099+0000 front 2024-03-26T04:53:54.245259+0000 (oldest deadline 2024-03-26T04:54:15.345177+0000) 2024-03-26T04:54:22.759 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:22 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:22.476+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:44.196054+0000 front 2024-03-26T04:53:44.196039+0000 (oldest deadline 2024-03-26T04:54:10.095928+0000) 2024-03-26T04:54:22.759 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:22 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:22.476+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:53.597025+0000 front 2024-03-26T04:53:53.597059+0000 (oldest deadline 2024-03-26T04:54:18.197443+0000) 2024-03-26T04:54:22.759 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:22 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:22.628+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.143968+0000 front 2024-03-26T04:53:46.143727+0000 (oldest deadline 2024-03-26T04:54:09.043713+0000) 2024-03-26T04:54:22.759 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:22 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:22.628+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:54.245099+0000 front 2024-03-26T04:53:54.245259+0000 (oldest deadline 2024-03-26T04:54:15.345177+0000) 2024-03-26T04:54:23.036 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 26 04:54:22 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-5[44571]: 2024-03-26T04:54:22.763+0000 7fd005367700 -1 osd.5 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.276691+0000 front 2024-03-26T04:53:46.276765+0000 (oldest deadline 2024-03-26T04:54:09.677086+0000) 2024-03-26T04:54:23.037 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 26 04:54:22 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-5[44571]: 2024-03-26T04:54:22.763+0000 7fd005367700 -1 osd.5 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:53.678558+0000 front 2024-03-26T04:53:53.678535+0000 (oldest deadline 2024-03-26T04:54:18.279041+0000) 2024-03-26T04:54:23.782 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:23 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:23.496+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:44.196054+0000 front 2024-03-26T04:53:44.196039+0000 (oldest deadline 2024-03-26T04:54:10.095928+0000) 2024-03-26T04:54:23.783 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:23 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:23.496+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:53.597025+0000 front 2024-03-26T04:53:53.597059+0000 (oldest deadline 2024-03-26T04:54:18.197443+0000) 2024-03-26T04:54:23.783 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:23 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:23.634+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.143968+0000 front 2024-03-26T04:53:46.143727+0000 (oldest deadline 2024-03-26T04:54:09.043713+0000) 2024-03-26T04:54:23.783 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:23 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:23.634+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:54.245099+0000 front 2024-03-26T04:53:54.245259+0000 (oldest deadline 2024-03-26T04:54:15.345177+0000) 2024-03-26T04:54:24.036 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 26 04:54:23 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-5[44571]: 2024-03-26T04:54:23.787+0000 7fd005367700 -1 osd.5 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.276691+0000 front 2024-03-26T04:53:46.276765+0000 (oldest deadline 2024-03-26T04:54:09.677086+0000) 2024-03-26T04:54:24.037 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 26 04:54:23 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-5[44571]: 2024-03-26T04:54:23.787+0000 7fd005367700 -1 osd.5 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:53.678558+0000 front 2024-03-26T04:53:53.678535+0000 (oldest deadline 2024-03-26T04:54:18.279041+0000) 2024-03-26T04:54:24.787 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:24 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:24.471+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:44.196054+0000 front 2024-03-26T04:53:44.196039+0000 (oldest deadline 2024-03-26T04:54:10.095928+0000) 2024-03-26T04:54:24.787 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:24 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:24.471+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:53.597025+0000 front 2024-03-26T04:53:53.597059+0000 (oldest deadline 2024-03-26T04:54:18.197443+0000) 2024-03-26T04:54:24.787 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:24 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:24.648+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.143968+0000 front 2024-03-26T04:53:46.143727+0000 (oldest deadline 2024-03-26T04:54:09.043713+0000) 2024-03-26T04:54:24.787 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:24 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:24.648+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:54.245099+0000 front 2024-03-26T04:53:54.245259+0000 (oldest deadline 2024-03-26T04:54:15.345177+0000) 2024-03-26T04:54:25.286 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 26 04:54:24 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-5[44571]: 2024-03-26T04:54:24.830+0000 7fd005367700 -1 osd.5 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.276691+0000 front 2024-03-26T04:53:46.276765+0000 (oldest deadline 2024-03-26T04:54:09.677086+0000) 2024-03-26T04:54:25.287 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 26 04:54:24 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-5[44571]: 2024-03-26T04:54:24.830+0000 7fd005367700 -1 osd.5 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:53.678558+0000 front 2024-03-26T04:53:53.678535+0000 (oldest deadline 2024-03-26T04:54:18.279041+0000) 2024-03-26T04:54:25.787 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:25 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:25.660+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.143968+0000 front 2024-03-26T04:53:46.143727+0000 (oldest deadline 2024-03-26T04:54:09.043713+0000) 2024-03-26T04:54:25.787 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:25 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:25.660+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:54.245099+0000 front 2024-03-26T04:53:54.245259+0000 (oldest deadline 2024-03-26T04:54:15.345177+0000) 2024-03-26T04:54:25.787 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:25 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:25.660+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6811 osd.2 since back 2024-03-26T04:54:00.046231+0000 front 2024-03-26T04:54:00.046202+0000 (oldest deadline 2024-03-26T04:54:24.746200+0000) 2024-03-26T04:54:25.787 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:25 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:25.460+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:44.196054+0000 front 2024-03-26T04:53:44.196039+0000 (oldest deadline 2024-03-26T04:54:10.095928+0000) 2024-03-26T04:54:25.787 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:25 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:25.460+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:53.597025+0000 front 2024-03-26T04:53:53.597059+0000 (oldest deadline 2024-03-26T04:54:18.197443+0000) 2024-03-26T04:54:25.787 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:25 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:25.460+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6811 osd.2 since back 2024-03-26T04:53:59.298440+0000 front 2024-03-26T04:53:59.298405+0000 (oldest deadline 2024-03-26T04:54:25.198421+0000) 2024-03-26T04:54:26.287 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 26 04:54:25 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-5[44571]: 2024-03-26T04:54:25.872+0000 7fd005367700 -1 osd.5 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.276691+0000 front 2024-03-26T04:53:46.276765+0000 (oldest deadline 2024-03-26T04:54:09.677086+0000) 2024-03-26T04:54:26.287 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 26 04:54:25 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-5[44571]: 2024-03-26T04:54:25.872+0000 7fd005367700 -1 osd.5 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:53.678558+0000 front 2024-03-26T04:53:53.678535+0000 (oldest deadline 2024-03-26T04:54:18.279041+0000) 2024-03-26T04:54:26.787 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:26 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:26.653+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.143968+0000 front 2024-03-26T04:53:46.143727+0000 (oldest deadline 2024-03-26T04:54:09.043713+0000) 2024-03-26T04:54:26.787 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:26 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:26.653+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:54.245099+0000 front 2024-03-26T04:53:54.245259+0000 (oldest deadline 2024-03-26T04:54:15.345177+0000) 2024-03-26T04:54:26.787 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:26 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:26.653+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6811 osd.2 since back 2024-03-26T04:54:00.046231+0000 front 2024-03-26T04:54:00.046202+0000 (oldest deadline 2024-03-26T04:54:24.746200+0000) 2024-03-26T04:54:26.787 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:26 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:26.455+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:44.196054+0000 front 2024-03-26T04:53:44.196039+0000 (oldest deadline 2024-03-26T04:54:10.095928+0000) 2024-03-26T04:54:26.787 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:26 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:26.455+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:53.597025+0000 front 2024-03-26T04:53:53.597059+0000 (oldest deadline 2024-03-26T04:54:18.197443+0000) 2024-03-26T04:54:26.787 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:26 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:26.455+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6811 osd.2 since back 2024-03-26T04:53:59.298440+0000 front 2024-03-26T04:53:59.298405+0000 (oldest deadline 2024-03-26T04:54:25.198421+0000) 2024-03-26T04:54:27.076 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 26 04:54:26 smithi183 podman[76264]: 2024-03-26 04:54:26.795680168 +0000 UTC m=+5.480548573 container died 9aa5e16229d1c85d8f8efa79f66083ff2df5b8c8ad6edef5937da5acb76b34a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-26T04:54:27.451 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 26 04:54:27 smithi183 podman[76264]: 2024-03-26 04:54:27.171560876 +0000 UTC m=+5.856429282 container cleanup 9aa5e16229d1c85d8f8efa79f66083ff2df5b8c8ad6edef5937da5acb76b34a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-5, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-26T04:54:27.451 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 26 04:54:27 smithi183 bash[76264]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-5 2024-03-26T04:54:27.787 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 26 04:54:27 smithi183 podman[76282]: 2024-03-26 04:54:27.456233686 +0000 UTC m=+0.652171280 container remove 9aa5e16229d1c85d8f8efa79f66083ff2df5b8c8ad6edef5937da5acb76b34a7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, io.buildah.version=1.33.5) 2024-03-26T04:54:27.787 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:27 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:27.458+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:44.196054+0000 front 2024-03-26T04:53:44.196039+0000 (oldest deadline 2024-03-26T04:54:10.095928+0000) 2024-03-26T04:54:27.787 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:27 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:27.458+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:53.597025+0000 front 2024-03-26T04:53:53.597059+0000 (oldest deadline 2024-03-26T04:54:18.197443+0000) 2024-03-26T04:54:27.787 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:27 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:27.458+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6811 osd.2 since back 2024-03-26T04:53:59.298440+0000 front 2024-03-26T04:53:59.298405+0000 (oldest deadline 2024-03-26T04:54:25.198421+0000) 2024-03-26T04:54:27.787 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:27 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:27.692+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.143968+0000 front 2024-03-26T04:53:46.143727+0000 (oldest deadline 2024-03-26T04:54:09.043713+0000) 2024-03-26T04:54:27.787 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:27 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:27.692+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:54.245099+0000 front 2024-03-26T04:53:54.245259+0000 (oldest deadline 2024-03-26T04:54:15.345177+0000) 2024-03-26T04:54:27.788 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:27 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:27.692+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6811 osd.2 since back 2024-03-26T04:54:00.046231+0000 front 2024-03-26T04:54:00.046202+0000 (oldest deadline 2024-03-26T04:54:24.746200+0000) 2024-03-26T04:54:28.130 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 26 04:54:27 smithi183 podman[76373]: 2024-03-26 04:54:27.696508132 +0000 UTC m=+0.018691231 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T04:54:28.130 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 26 04:54:27 smithi183 podman[76373]: 2024-03-26 04:54:27.798861482 +0000 UTC m=+0.121044580 container create 6ebf5b6df91ab21a0381da6878d0b8ec0b0c15265a3fb7437934fe697e1e53a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-5e70124) 2024-03-26T04:54:28.130 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 26 04:54:28 smithi183 podman[76373]: 2024-03-26 04:54:28.034682899 +0000 UTC m=+0.356865997 container init 6ebf5b6df91ab21a0381da6878d0b8ec0b0c15265a3fb7437934fe697e1e53a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319) 2024-03-26T04:54:28.130 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 26 04:54:28 smithi183 podman[76373]: 2024-03-26 04:54:28.037995123 +0000 UTC m=+0.360178222 container start 6ebf5b6df91ab21a0381da6878d0b8ec0b0c15265a3fb7437934fe697e1e53a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T04:54:28.450 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 26 04:54:28 smithi183 podman[76373]: 2024-03-26 04:54:28.134686752 +0000 UTC m=+0.456869851 container attach 6ebf5b6df91ab21a0381da6878d0b8ec0b0c15265a3fb7437934fe697e1e53a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, RELEASE=squid-5e70124) 2024-03-26T04:54:28.450 INFO:journalctl@ceph.osd.5.smithi183.stdout:Mar 26 04:54:28 smithi183 podman[76373]: 2024-03-26 04:54:28.327669772 +0000 UTC m=+0.649852873 container died 6ebf5b6df91ab21a0381da6878d0b8ec0b0c15265a3fb7437934fe697e1e53a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-5-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=squid-5e70124, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0) 2024-03-26T04:54:28.733 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:28 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:28.455+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:44.196054+0000 front 2024-03-26T04:53:44.196039+0000 (oldest deadline 2024-03-26T04:54:10.095928+0000) 2024-03-26T04:54:28.733 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:28 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:28.455+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:53.597025+0000 front 2024-03-26T04:53:53.597059+0000 (oldest deadline 2024-03-26T04:54:18.197443+0000) 2024-03-26T04:54:28.733 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:28 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:28.455+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6811 osd.2 since back 2024-03-26T04:53:59.298440+0000 front 2024-03-26T04:53:59.298405+0000 (oldest deadline 2024-03-26T04:54:25.198421+0000) 2024-03-26T04:54:28.885 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d7d59826-eb26-11ee-95d0-87774f69a715@osd.5.service' 2024-03-26T04:54:28.927 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:54:28.927 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-26T04:54:28.927 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-26T04:54:28.927 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-d7d59826-eb26-11ee-95d0-87774f69a715@osd.6 2024-03-26T04:54:28.990 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:28 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:28.738+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.143968+0000 front 2024-03-26T04:53:46.143727+0000 (oldest deadline 2024-03-26T04:54:09.043713+0000) 2024-03-26T04:54:28.990 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:28 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:28.738+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:54.245099+0000 front 2024-03-26T04:53:54.245259+0000 (oldest deadline 2024-03-26T04:54:15.345177+0000) 2024-03-26T04:54:28.990 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:28 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:28.738+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6811 osd.2 since back 2024-03-26T04:54:00.046231+0000 front 2024-03-26T04:54:00.046202+0000 (oldest deadline 2024-03-26T04:54:24.746200+0000) 2024-03-26T04:54:28.990 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:28 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:28.738+0000 7f37f2ef4700 -1 osd.7 822 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24466.0:2191 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e822) 2024-03-26T04:54:29.245 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:29 smithi183 systemd[1]: Stopping Ceph osd.6 for d7d59826-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:54:29.537 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:29 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:29.250+0000 7fcb6c1dc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:54:29.537 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:29 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:29.250+0000 7fcb6c1dc700 -1 osd.6 822 *** Got signal Terminated *** 2024-03-26T04:54:29.537 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:29 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:29.250+0000 7fcb6c1dc700 -1 osd.6 822 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T04:54:29.537 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:29 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:29.412+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:44.196054+0000 front 2024-03-26T04:53:44.196039+0000 (oldest deadline 2024-03-26T04:54:10.095928+0000) 2024-03-26T04:54:29.537 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:29 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:29.412+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:53.597025+0000 front 2024-03-26T04:53:53.597059+0000 (oldest deadline 2024-03-26T04:54:18.197443+0000) 2024-03-26T04:54:29.537 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:29 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:29.412+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6811 osd.2 since back 2024-03-26T04:53:59.298440+0000 front 2024-03-26T04:53:59.298405+0000 (oldest deadline 2024-03-26T04:54:25.198421+0000) 2024-03-26T04:54:30.036 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:29 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:29.705+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.143968+0000 front 2024-03-26T04:53:46.143727+0000 (oldest deadline 2024-03-26T04:54:09.043713+0000) 2024-03-26T04:54:30.037 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:29 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:29.705+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:54.245099+0000 front 2024-03-26T04:53:54.245259+0000 (oldest deadline 2024-03-26T04:54:15.345177+0000) 2024-03-26T04:54:30.037 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:29 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:29.705+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6811 osd.2 since back 2024-03-26T04:54:00.046231+0000 front 2024-03-26T04:54:00.046202+0000 (oldest deadline 2024-03-26T04:54:24.746200+0000) 2024-03-26T04:54:30.037 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:29 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:29.706+0000 7f37f2ef4700 -1 osd.7 822 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24466.0:2191 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e822) 2024-03-26T04:54:30.787 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:30 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:30.416+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:44.196054+0000 front 2024-03-26T04:53:44.196039+0000 (oldest deadline 2024-03-26T04:54:10.095928+0000) 2024-03-26T04:54:30.787 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:30 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:30.416+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:53.597025+0000 front 2024-03-26T04:53:53.597059+0000 (oldest deadline 2024-03-26T04:54:18.197443+0000) 2024-03-26T04:54:30.787 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:30 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:30.416+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6811 osd.2 since back 2024-03-26T04:53:59.298440+0000 front 2024-03-26T04:53:59.298405+0000 (oldest deadline 2024-03-26T04:54:25.198421+0000) 2024-03-26T04:54:30.787 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:30 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:30.660+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.143968+0000 front 2024-03-26T04:53:46.143727+0000 (oldest deadline 2024-03-26T04:54:09.043713+0000) 2024-03-26T04:54:30.787 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:30 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:30.660+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:54.245099+0000 front 2024-03-26T04:53:54.245259+0000 (oldest deadline 2024-03-26T04:54:15.345177+0000) 2024-03-26T04:54:30.787 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:30 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:30.660+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6811 osd.2 since back 2024-03-26T04:54:00.046231+0000 front 2024-03-26T04:54:00.046202+0000 (oldest deadline 2024-03-26T04:54:24.746200+0000) 2024-03-26T04:54:30.787 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:30 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:30.660+0000 7f37f2ef4700 -1 osd.7 822 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24466.0:2191 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e822) 2024-03-26T04:54:31.701 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:31 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:31.456+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:44.196054+0000 front 2024-03-26T04:53:44.196039+0000 (oldest deadline 2024-03-26T04:54:10.095928+0000) 2024-03-26T04:54:31.702 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:31 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:31.456+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:53.597025+0000 front 2024-03-26T04:53:53.597059+0000 (oldest deadline 2024-03-26T04:54:18.197443+0000) 2024-03-26T04:54:31.702 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:31 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:31.456+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6811 osd.2 since back 2024-03-26T04:53:59.298440+0000 front 2024-03-26T04:53:59.298405+0000 (oldest deadline 2024-03-26T04:54:25.198421+0000) 2024-03-26T04:54:32.036 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:31 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:31.705+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.143968+0000 front 2024-03-26T04:53:46.143727+0000 (oldest deadline 2024-03-26T04:54:09.043713+0000) 2024-03-26T04:54:32.036 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:31 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:31.705+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:54.245099+0000 front 2024-03-26T04:53:54.245259+0000 (oldest deadline 2024-03-26T04:54:15.345177+0000) 2024-03-26T04:54:32.037 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:31 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:31.705+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6811 osd.2 since back 2024-03-26T04:54:00.046231+0000 front 2024-03-26T04:54:00.046202+0000 (oldest deadline 2024-03-26T04:54:24.746200+0000) 2024-03-26T04:54:32.037 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:31 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:31.706+0000 7f37f2ef4700 -1 osd.7 822 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.24466.0:2191 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch in=5b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e822) 2024-03-26T04:54:32.786 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:32 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:32.439+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:44.196054+0000 front 2024-03-26T04:53:44.196039+0000 (oldest deadline 2024-03-26T04:54:10.095928+0000) 2024-03-26T04:54:32.787 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:32 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:32.439+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:53.597025+0000 front 2024-03-26T04:53:53.597059+0000 (oldest deadline 2024-03-26T04:54:18.197443+0000) 2024-03-26T04:54:32.787 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:32 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:32.439+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6811 osd.2 since back 2024-03-26T04:53:59.298440+0000 front 2024-03-26T04:53:59.298405+0000 (oldest deadline 2024-03-26T04:54:25.198421+0000) 2024-03-26T04:54:32.787 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:32 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:32.668+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.143968+0000 front 2024-03-26T04:53:46.143727+0000 (oldest deadline 2024-03-26T04:54:09.043713+0000) 2024-03-26T04:54:32.787 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:32 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:32.668+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:54.245099+0000 front 2024-03-26T04:53:54.245259+0000 (oldest deadline 2024-03-26T04:54:15.345177+0000) 2024-03-26T04:54:32.787 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:32 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:32.668+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6811 osd.2 since back 2024-03-26T04:54:00.046231+0000 front 2024-03-26T04:54:00.046202+0000 (oldest deadline 2024-03-26T04:54:24.746200+0000) 2024-03-26T04:54:33.786 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:33 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:33.676+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.143968+0000 front 2024-03-26T04:53:46.143727+0000 (oldest deadline 2024-03-26T04:54:09.043713+0000) 2024-03-26T04:54:33.787 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:33 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:33.676+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:54.245099+0000 front 2024-03-26T04:53:54.245259+0000 (oldest deadline 2024-03-26T04:54:15.345177+0000) 2024-03-26T04:54:33.787 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:33 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:33.676+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6811 osd.2 since back 2024-03-26T04:54:00.046231+0000 front 2024-03-26T04:54:00.046202+0000 (oldest deadline 2024-03-26T04:54:24.746200+0000) 2024-03-26T04:54:33.787 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:33 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:33.461+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:44.196054+0000 front 2024-03-26T04:53:44.196039+0000 (oldest deadline 2024-03-26T04:54:10.095928+0000) 2024-03-26T04:54:33.787 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:33 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:33.461+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:53.597025+0000 front 2024-03-26T04:53:53.597059+0000 (oldest deadline 2024-03-26T04:54:18.197443+0000) 2024-03-26T04:54:33.787 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:33 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6[50116]: 2024-03-26T04:54:33.461+0000 7fcb647c8700 -1 osd.6 822 heartbeat_check: no reply from 172.21.15.16:6811 osd.2 since back 2024-03-26T04:53:59.298440+0000 front 2024-03-26T04:53:59.298405+0000 (oldest deadline 2024-03-26T04:54:25.198421+0000) 2024-03-26T04:54:34.861 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:34 smithi183 podman[76480]: 2024-03-26 04:54:34.587128311 +0000 UTC m=+5.430616372 container died 3b5d1db6c68e6502d965003d437cbdcafcf4fa76fe6985c8a19e5cf80cbea1a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-26T04:54:34.861 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:34 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:34.665+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.143968+0000 front 2024-03-26T04:53:46.143727+0000 (oldest deadline 2024-03-26T04:54:09.043713+0000) 2024-03-26T04:54:34.861 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:34 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:34.665+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:54.245099+0000 front 2024-03-26T04:53:54.245259+0000 (oldest deadline 2024-03-26T04:54:15.345177+0000) 2024-03-26T04:54:34.861 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:34 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:34.665+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6811 osd.2 since back 2024-03-26T04:54:00.046231+0000 front 2024-03-26T04:54:00.046202+0000 (oldest deadline 2024-03-26T04:54:24.746200+0000) 2024-03-26T04:54:35.194 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:34 smithi183 podman[76480]: 2024-03-26 04:54:34.948827223 +0000 UTC m=+5.792315300 container cleanup 3b5d1db6c68e6502d965003d437cbdcafcf4fa76fe6985c8a19e5cf80cbea1a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319) 2024-03-26T04:54:35.194 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:34 smithi183 bash[76480]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6 2024-03-26T04:54:35.501 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:35 smithi183 podman[76498]: 2024-03-26 04:54:35.200062706 +0000 UTC m=+0.608348722 container remove 3b5d1db6c68e6502d965003d437cbdcafcf4fa76fe6985c8a19e5cf80cbea1a8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6, ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, GIT_CLEAN=True) 2024-03-26T04:54:35.787 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:35 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:35.640+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.143968+0000 front 2024-03-26T04:53:46.143727+0000 (oldest deadline 2024-03-26T04:54:09.043713+0000) 2024-03-26T04:54:35.787 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:35 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:35.640+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:54.245099+0000 front 2024-03-26T04:53:54.245259+0000 (oldest deadline 2024-03-26T04:54:15.345177+0000) 2024-03-26T04:54:35.787 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:35 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:35.640+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6811 osd.2 since back 2024-03-26T04:54:00.046231+0000 front 2024-03-26T04:54:00.046202+0000 (oldest deadline 2024-03-26T04:54:24.746200+0000) 2024-03-26T04:54:35.787 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:35 smithi183 podman[76590]: 2024-03-26 04:54:35.409367333 +0000 UTC m=+0.017122199 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T04:54:35.787 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:35 smithi183 podman[76590]: 2024-03-26 04:54:35.509256483 +0000 UTC m=+0.117011386 container create c07b691640450611132e4030e07e0608b313c584a83b3b45a4646570b22e1f84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-26T04:54:35.787 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:35 smithi183 podman[76590]: 2024-03-26 04:54:35.728579707 +0000 UTC m=+0.336334568 container init c07b691640450611132e4030e07e0608b313c584a83b3b45a4646570b22e1f84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-26T04:54:35.787 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:35 smithi183 podman[76590]: 2024-03-26 04:54:35.731616737 +0000 UTC m=+0.339371597 container start c07b691640450611132e4030e07e0608b313c584a83b3b45a4646570b22e1f84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-5e70124, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-26T04:54:36.286 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:35 smithi183 podman[76590]: 2024-03-26 04:54:35.81179645 +0000 UTC m=+0.419551322 container attach c07b691640450611132e4030e07e0608b313c584a83b3b45a4646570b22e1f84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T04:54:36.287 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:36 smithi183 podman[76590]: 2024-03-26 04:54:36.023928454 +0000 UTC m=+0.631683313 container died c07b691640450611132e4030e07e0608b313c584a83b3b45a4646570b22e1f84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-26T04:54:36.593 INFO:journalctl@ceph.osd.6.smithi183.stdout:Mar 26 04:54:36 smithi183 podman[76615]: 2024-03-26 04:54:36.573582328 +0000 UTC m=+0.540305821 container remove c07b691640450611132e4030e07e0608b313c584a83b3b45a4646570b22e1f84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-26T04:54:36.597 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d7d59826-eb26-11ee-95d0-87774f69a715@osd.6.service' 2024-03-26T04:54:36.678 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:54:36.679 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-26T04:54:36.679 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-26T04:54:36.679 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-d7d59826-eb26-11ee-95d0-87774f69a715@osd.7 2024-03-26T04:54:36.980 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:36 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:36.660+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.143968+0000 front 2024-03-26T04:53:46.143727+0000 (oldest deadline 2024-03-26T04:54:09.043713+0000) 2024-03-26T04:54:36.980 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:36 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:36.660+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:54.245099+0000 front 2024-03-26T04:53:54.245259+0000 (oldest deadline 2024-03-26T04:54:15.345177+0000) 2024-03-26T04:54:36.980 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:36 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:36.660+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6811 osd.2 since back 2024-03-26T04:54:00.046231+0000 front 2024-03-26T04:54:00.046202+0000 (oldest deadline 2024-03-26T04:54:24.746200+0000) 2024-03-26T04:54:36.980 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:36 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:36.660+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6815 osd.3 since back 2024-03-26T04:54:09.347525+0000 front 2024-03-26T04:54:09.347581+0000 (oldest deadline 2024-03-26T04:54:35.747902+0000) 2024-03-26T04:54:36.980 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:36 smithi183 systemd[1]: Stopping Ceph osd.7 for d7d59826-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:54:36.980 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:36 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:36.984+0000 7f37fa908700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:54:36.980 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:36 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:36.984+0000 7f37fa908700 -1 osd.7 822 *** Got signal Terminated *** 2024-03-26T04:54:36.981 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:36 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:36.984+0000 7f37fa908700 -1 osd.7 822 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-26T04:54:38.037 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:37 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:37.659+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.143968+0000 front 2024-03-26T04:53:46.143727+0000 (oldest deadline 2024-03-26T04:54:09.043713+0000) 2024-03-26T04:54:38.037 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:37 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:37.659+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:54.245099+0000 front 2024-03-26T04:53:54.245259+0000 (oldest deadline 2024-03-26T04:54:15.345177+0000) 2024-03-26T04:54:38.037 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:37 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:37.659+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6811 osd.2 since back 2024-03-26T04:54:00.046231+0000 front 2024-03-26T04:54:00.046202+0000 (oldest deadline 2024-03-26T04:54:24.746200+0000) 2024-03-26T04:54:38.037 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:37 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:37.659+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6815 osd.3 since back 2024-03-26T04:54:09.347525+0000 front 2024-03-26T04:54:09.347581+0000 (oldest deadline 2024-03-26T04:54:35.747902+0000) 2024-03-26T04:54:39.036 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:38 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:38.672+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.143968+0000 front 2024-03-26T04:53:46.143727+0000 (oldest deadline 2024-03-26T04:54:09.043713+0000) 2024-03-26T04:54:39.037 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:38 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:38.672+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:54.245099+0000 front 2024-03-26T04:53:54.245259+0000 (oldest deadline 2024-03-26T04:54:15.345177+0000) 2024-03-26T04:54:39.037 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:38 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:38.672+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6811 osd.2 since back 2024-03-26T04:54:00.046231+0000 front 2024-03-26T04:54:00.046202+0000 (oldest deadline 2024-03-26T04:54:24.746200+0000) 2024-03-26T04:54:39.037 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:38 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:38.672+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6815 osd.3 since back 2024-03-26T04:54:09.347525+0000 front 2024-03-26T04:54:09.347581+0000 (oldest deadline 2024-03-26T04:54:35.747902+0000) 2024-03-26T04:54:40.036 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:39 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:39.693+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.143968+0000 front 2024-03-26T04:53:46.143727+0000 (oldest deadline 2024-03-26T04:54:09.043713+0000) 2024-03-26T04:54:40.037 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:39 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:39.693+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:54.245099+0000 front 2024-03-26T04:53:54.245259+0000 (oldest deadline 2024-03-26T04:54:15.345177+0000) 2024-03-26T04:54:40.037 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:39 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:39.693+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6811 osd.2 since back 2024-03-26T04:54:00.046231+0000 front 2024-03-26T04:54:00.046202+0000 (oldest deadline 2024-03-26T04:54:24.746200+0000) 2024-03-26T04:54:40.037 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:39 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:39.693+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6815 osd.3 since back 2024-03-26T04:54:09.347525+0000 front 2024-03-26T04:54:09.347581+0000 (oldest deadline 2024-03-26T04:54:35.747902+0000) 2024-03-26T04:54:40.037 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:39 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:39.693+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.183:6802 osd.4 since back 2024-03-26T04:54:15.748384+0000 front 2024-03-26T04:54:15.748301+0000 (oldest deadline 2024-03-26T04:54:39.248275+0000) 2024-03-26T04:54:41.037 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:40 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:40.695+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.143968+0000 front 2024-03-26T04:53:46.143727+0000 (oldest deadline 2024-03-26T04:54:09.043713+0000) 2024-03-26T04:54:41.037 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:40 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:40.695+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:54.245099+0000 front 2024-03-26T04:53:54.245259+0000 (oldest deadline 2024-03-26T04:54:15.345177+0000) 2024-03-26T04:54:41.037 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:40 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:40.695+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6811 osd.2 since back 2024-03-26T04:54:00.046231+0000 front 2024-03-26T04:54:00.046202+0000 (oldest deadline 2024-03-26T04:54:24.746200+0000) 2024-03-26T04:54:41.037 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:40 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:40.695+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6815 osd.3 since back 2024-03-26T04:54:09.347525+0000 front 2024-03-26T04:54:09.347581+0000 (oldest deadline 2024-03-26T04:54:35.747902+0000) 2024-03-26T04:54:41.037 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:40 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:40.695+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.183:6802 osd.4 since back 2024-03-26T04:54:15.748384+0000 front 2024-03-26T04:54:15.748301+0000 (oldest deadline 2024-03-26T04:54:39.248275+0000) 2024-03-26T04:54:42.037 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:41 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:41.649+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6803 osd.0 since back 2024-03-26T04:53:46.143968+0000 front 2024-03-26T04:53:46.143727+0000 (oldest deadline 2024-03-26T04:54:09.043713+0000) 2024-03-26T04:54:42.037 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:41 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:41.649+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6807 osd.1 since back 2024-03-26T04:53:54.245099+0000 front 2024-03-26T04:53:54.245259+0000 (oldest deadline 2024-03-26T04:54:15.345177+0000) 2024-03-26T04:54:42.037 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:41 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:41.649+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6811 osd.2 since back 2024-03-26T04:54:00.046231+0000 front 2024-03-26T04:54:00.046202+0000 (oldest deadline 2024-03-26T04:54:24.746200+0000) 2024-03-26T04:54:42.037 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:41 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:41.649+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.16:6815 osd.3 since back 2024-03-26T04:54:09.347525+0000 front 2024-03-26T04:54:09.347581+0000 (oldest deadline 2024-03-26T04:54:35.747902+0000) 2024-03-26T04:54:42.037 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:41 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7[55709]: 2024-03-26T04:54:41.649+0000 7f37f2ef4700 -1 osd.7 822 heartbeat_check: no reply from 172.21.15.183:6802 osd.4 since back 2024-03-26T04:54:15.748384+0000 front 2024-03-26T04:54:15.748301+0000 (oldest deadline 2024-03-26T04:54:39.248275+0000) 2024-03-26T04:54:42.596 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:42 smithi183 podman[76693]: 2024-03-26 04:54:42.331469898 +0000 UTC m=+5.428454716 container died fbe26e11429c63f756407a3f912d3e029c3c2eeb264e9fab310cbeaa3990b996 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-26T04:54:42.961 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:42 smithi183 podman[76693]: 2024-03-26 04:54:42.692585321 +0000 UTC m=+5.789570133 container cleanup fbe26e11429c63f756407a3f912d3e029c3c2eeb264e9fab310cbeaa3990b996 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-26T04:54:42.961 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:42 smithi183 bash[76693]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7 2024-03-26T04:54:43.286 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:42 smithi183 podman[76722]: 2024-03-26 04:54:42.966962127 +0000 UTC m=+0.630049179 container remove fbe26e11429c63f756407a3f912d3e029c3c2eeb264e9fab310cbeaa3990b996 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7, RELEASE=squid-5e70124, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-26T04:54:43.581 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:43 smithi183 podman[76815]: 2024-03-26 04:54:43.204939472 +0000 UTC m=+0.017640380 image pull 5f21c926413668ef2178e7fc64376b7950a84a779ce3303236a4c41ae3e83767 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2 2024-03-26T04:54:43.582 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:43 smithi183 podman[76815]: 2024-03-26 04:54:43.309420337 +0000 UTC m=+0.122121286 container create 0ab1333a99df799ec0ddf0b31dbae5be3a7a29b3796653f03daddbbb8ca2b348 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7-deactivate, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-26T04:54:43.582 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:43 smithi183 podman[76815]: 2024-03-26 04:54:43.512048936 +0000 UTC m=+0.324749843 container init 0ab1333a99df799ec0ddf0b31dbae5be3a7a29b3796653f03daddbbb8ca2b348 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.5, RELEASE=squid-5e70124, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-26T04:54:43.582 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:43 smithi183 podman[76815]: 2024-03-26 04:54:43.515074291 +0000 UTC m=+0.327775206 container start 0ab1333a99df799ec0ddf0b31dbae5be3a7a29b3796653f03daddbbb8ca2b348 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-5e70124, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-26T04:54:43.582 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:43 smithi183 podman[76815]: 2024-03-26 04:54:43.587032739 +0000 UTC m=+0.399733658 container attach 0ab1333a99df799ec0ddf0b31dbae5be3a7a29b3796653f03daddbbb8ca2b348 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-26T04:54:44.036 INFO:journalctl@ceph.osd.7.smithi183.stdout:Mar 26 04:54:43 smithi183 podman[76815]: 2024-03-26 04:54:43.811584347 +0000 UTC m=+0.624285253 container died 0ab1333a99df799ec0ddf0b31dbae5be3a7a29b3796653f03daddbbb8ca2b348 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-osd-7-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-5e70124, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-26T04:54:44.426 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d7d59826-eb26-11ee-95d0-87774f69a715@osd.7.service' 2024-03-26T04:54:44.469 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:54:44.469 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-26T04:54:44.469 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-26T04:54:44.469 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ceph-d7d59826-eb26-11ee-95d0-87774f69a715@rgw.foo.a 2024-03-26T04:54:44.786 INFO:journalctl@ceph.rgw.foo.a.smithi016.stdout:Mar 26 04:54:44 smithi016 systemd[1]: Stopping Ceph rgw.foo.a for d7d59826-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:54:45.128 INFO:journalctl@ceph.rgw.foo.a.smithi016.stdout:Mar 26 04:54:44 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-rgw-foo-a[63917]: 2024-03-26T04:54:44.790+0000 7fb58b13b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-26T04:54:45.128 INFO:journalctl@ceph.rgw.foo.a.smithi016.stdout:Mar 26 04:54:44 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-rgw-foo-a[63917]: 2024-03-26T04:54:44.790+0000 7fb5999b8ac0 -1 shutting down 2024-03-26T04:54:55.078 INFO:journalctl@ceph.rgw.foo.a.smithi016.stdout:Mar 26 04:54:54 smithi016 bash[153183]: time="2024-03-26T04:54:54Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-d7d59826-eb26-11ee-95d0-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-26T04:54:55.078 INFO:journalctl@ceph.rgw.foo.a.smithi016.stdout:Mar 26 04:54:54 smithi016 podman[153183]: 2024-03-26 04:54:54.809488457 +0000 UTC m=+10.134019634 container died df20b4bd81a97ca97f72f6895b3614c0e8d150782cebd68c166eb08cabb40ca8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-rgw-foo-a, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-5e70124) 2024-03-26T04:54:55.344 INFO:journalctl@ceph.rgw.foo.a.smithi016.stdout:Mar 26 04:54:55 smithi016 podman[153183]: 2024-03-26 04:54:55.156811345 +0000 UTC m=+10.481342522 container cleanup df20b4bd81a97ca97f72f6895b3614c0e8d150782cebd68c166eb08cabb40ca8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-rgw-foo-a, RELEASE=squid-5e70124, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-26T04:54:55.344 INFO:journalctl@ceph.rgw.foo.a.smithi016.stdout:Mar 26 04:54:55 smithi016 bash[153183]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715-rgw-foo-a 2024-03-26T04:54:55.590 DEBUG:teuthology.orchestra.run.smithi016:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d7d59826-eb26-11ee-95d0-87774f69a715@rgw.foo.a.service' 2024-03-26T04:54:55.615 INFO:journalctl@ceph.rgw.foo.a.smithi016.stdout:Mar 26 04:54:55 smithi016 podman[153210]: 2024-03-26 04:54:55.349401621 +0000 UTC m=+0.532261995 container remove df20b4bd81a97ca97f72f6895b3614c0e8d150782cebd68c166eb08cabb40ca8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-rgw-foo-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-5e70124, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-26T04:54:55.615 INFO:journalctl@ceph.rgw.foo.a.smithi016.stdout:Mar 26 04:54:55 smithi016 systemd[1]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715@rgw.foo.a.service: Main process exited, code=exited, status=137/n/a 2024-03-26T04:54:55.615 INFO:journalctl@ceph.rgw.foo.a.smithi016.stdout:Mar 26 04:54:55 smithi016 systemd[1]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715@rgw.foo.a.service: Failed with result 'exit-code'. 2024-03-26T04:54:55.615 INFO:journalctl@ceph.rgw.foo.a.smithi016.stdout:Mar 26 04:54:55 smithi016 systemd[1]: Stopped Ceph rgw.foo.a for d7d59826-eb26-11ee-95d0-87774f69a715. 2024-03-26T04:54:55.616 INFO:journalctl@ceph.rgw.foo.a.smithi016.stdout:Mar 26 04:54:55 smithi016 systemd[1]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715@rgw.foo.a.service: Consumed 6.049s CPU time. 2024-03-26T04:54:55.630 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:54:55.630 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-26T04:54:55.630 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-26T04:54:55.631 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-d7d59826-eb26-11ee-95d0-87774f69a715@prometheus.a 2024-03-26T04:54:55.928 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:54:55 smithi183 systemd[1]: Stopping Ceph prometheus.a for d7d59826-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:54:56.244 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:54:55 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[65252]: ts=2024-03-26T04:54:55.933Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-26T04:54:56.244 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:54:55 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[65252]: ts=2024-03-26T04:54:55.933Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-26T04:54:56.244 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:54:55 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[65252]: ts=2024-03-26T04:54:55.933Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-26T04:54:56.244 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:54:55 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[65252]: ts=2024-03-26T04:54:55.933Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-26T04:54:56.244 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:54:55 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[65252]: ts=2024-03-26T04:54:55.933Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-26T04:54:56.244 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:54:55 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[65252]: ts=2024-03-26T04:54:55.933Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-26T04:54:56.244 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:54:55 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[65252]: ts=2024-03-26T04:54:55.933Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-26T04:54:56.245 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:54:55 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[65252]: ts=2024-03-26T04:54:55.933Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-26T04:54:56.245 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:54:55 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[65252]: ts=2024-03-26T04:54:55.934Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-26T04:54:56.245 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:54:55 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[65252]: ts=2024-03-26T04:54:55.934Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-26T04:54:56.245 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:54:55 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[65252]: ts=2024-03-26T04:54:55.934Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-26T04:54:56.245 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:54:55 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a[65252]: ts=2024-03-26T04:54:55.934Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-26T04:54:56.245 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:54:55 smithi183 podman[76920]: 2024-03-26 04:54:55.943837122 +0000 UTC m=+0.111675462 container died 12fe18d01c61267dee1c28e0932f6235cdadea946603d73e6fcfbee503e8c905 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:54:56.245 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:54:56 smithi183 podman[76920]: 2024-03-26 04:54:56.249586923 +0000 UTC m=+0.417425262 container cleanup 12fe18d01c61267dee1c28e0932f6235cdadea946603d73e6fcfbee503e8c905 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:54:56.245 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:54:56 smithi183 bash[76920]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a 2024-03-26T04:54:56.536 INFO:journalctl@ceph.prometheus.a.smithi183.stdout:Mar 26 04:54:56 smithi183 podman[76934]: 2024-03-26 04:54:56.484292023 +0000 UTC m=+0.542338933 container remove 12fe18d01c61267dee1c28e0932f6235cdadea946603d73e6fcfbee503e8c905 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-26T04:54:56.711 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d7d59826-eb26-11ee-95d0-87774f69a715@prometheus.a.service' 2024-03-26T04:54:56.748 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-26T04:54:56.749 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-26T04:54:56.749 DEBUG:teuthology.orchestra.run.smithi016:> sudo cephadm rm-cluster --fsid d7d59826-eb26-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-26T04:54:56.945 INFO:teuthology.orchestra.run.smithi016.stdout:Deleting cluster with fsid: d7d59826-eb26-11ee-95d0-87774f69a715 2024-03-26T04:54:58.657 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:54:58 smithi016 systemd[1]: Stopping Ceph alertmanager.a for d7d59826-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:54:59.091 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:54:58 smithi016 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-alertmanager-a[73529]: ts=2024-03-26T04:54:58.787Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-26T04:54:59.091 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:54:58 smithi016 podman[153688]: 2024-03-26 04:54:58.828218433 +0000 UTC m=+0.135227455 container died 6550a87ba310b28d3bb3e5d96eb439b32ddc74613365f608aeb975cb0873a1a4 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:54:59.407 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:54:59 smithi016 podman[153688]: 2024-03-26 04:54:59.187186704 +0000 UTC m=+0.494195726 container cleanup 6550a87ba310b28d3bb3e5d96eb439b32ddc74613365f608aeb975cb0873a1a4 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:54:59.407 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:54:59 smithi016 bash[153688]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715-alertmanager-a 2024-03-26T04:54:59.848 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:54:59 smithi016 podman[153702]: 2024-03-26 04:54:59.438260547 +0000 UTC m=+0.601674120 container remove 6550a87ba310b28d3bb3e5d96eb439b32ddc74613365f608aeb975cb0873a1a4 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-26T04:54:59.848 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:54:59 smithi016 podman[153702]: 2024-03-26 04:54:59.505396984 +0000 UTC m=+0.668810558 volume remove 38d4389a60992805c767df881a5fdd32ba1f98f9f3dd0818c1b010de25eb6c1e 2024-03-26T04:54:59.848 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:54:59 smithi016 systemd[1]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-26T04:54:59.849 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:54:59 smithi016 systemd[1]: Stopped Ceph alertmanager.a for d7d59826-eb26-11ee-95d0-87774f69a715. 2024-03-26T04:54:59.849 INFO:journalctl@ceph.alertmanager.a.smithi016.stdout:Mar 26 04:54:59 smithi016 systemd[1]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715@alertmanager.a.service: Consumed 2.325s CPU time. 2024-03-26T04:55:00.157 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:54:59 smithi016 systemd[1]: Stopping Ceph node-exporter.a for d7d59826-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:55:00.157 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:55:00 smithi016 podman[153814]: 2024-03-26 04:55:00.102113079 +0000 UTC m=+0.110742703 container died 115b83f6ff7c5286859de9598beeb2047f7f14d4ca13f8c5db5c4f394f28ea0b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T04:55:00.657 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:55:00 smithi016 podman[153814]: 2024-03-26 04:55:00.474750447 +0000 UTC m=+0.483380121 container cleanup 115b83f6ff7c5286859de9598beeb2047f7f14d4ca13f8c5db5c4f394f28ea0b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T04:55:00.657 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:55:00 smithi016 bash[153814]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a 2024-03-26T04:55:01.041 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:55:00 smithi016 podman[153828]: 2024-03-26 04:55:00.784335846 +0000 UTC m=+0.686757057 container remove 115b83f6ff7c5286859de9598beeb2047f7f14d4ca13f8c5db5c4f394f28ea0b (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-26T04:55:01.041 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:55:00 smithi016 systemd[1]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-26T04:55:01.407 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:55:01 smithi016 systemd[1]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-26T04:55:01.407 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:55:01 smithi016 systemd[1]: Stopped Ceph node-exporter.a for d7d59826-eb26-11ee-95d0-87774f69a715. 2024-03-26T04:55:01.407 INFO:journalctl@ceph.node-exporter.a.smithi016.stdout:Mar 26 04:55:01 smithi016 systemd[1]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715@node-exporter.a.service: Consumed 5.127s CPU time. 2024-03-26T04:55:01.665 DEBUG:teuthology.orchestra.run.smithi183:> sudo cephadm rm-cluster --fsid d7d59826-eb26-11ee-95d0-87774f69a715 --force --keep-logs 2024-03-26T04:55:01.858 INFO:teuthology.orchestra.run.smithi183.stdout:Deleting cluster with fsid: d7d59826-eb26-11ee-95d0-87774f69a715 2024-03-26T04:55:03.286 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:55:03 smithi183 systemd[1]: Stopping Ceph iscsi.iscsi.a for d7d59826-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:55:03.786 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:55:03 smithi183 ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a[60505]: debug Shutdown received 2024-03-26T04:55:13.671 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:55:13 smithi183 bash[77334]: time="2024-03-26T04:55:13Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-03-26T04:55:13.671 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:55:13 smithi183 podman[77334]: 2024-03-26 04:55:13.414778076 +0000 UTC m=+10.128535295 container died 77cff6840666b29ce345ea3aae672ce5d92c60e7517817ffd6ca3b6ad447cc2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-26T04:55:13.671 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:55:13 smithi183 podman[77334]: 2024-03-26 04:55:13.677181654 +0000 UTC m=+10.390938875 container cleanup 77cff6840666b29ce345ea3aae672ce5d92c60e7517817ffd6ca3b6ad447cc2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-5e70124, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-26T04:55:13.672 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:55:13 smithi183 bash[77334]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a 2024-03-26T04:55:14.286 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:55:13 smithi183 podman[77348]: 2024-03-26 04:55:13.936646533 +0000 UTC m=+0.522281672 container remove 77cff6840666b29ce345ea3aae672ce5d92c60e7517817ffd6ca3b6ad447cc2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:0006c6da31a166418cf302d3d12786a1c8582483d6158cbc13fc4d4e834fddf2, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-iscsi-iscsi-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-5e70124, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319) 2024-03-26T04:55:14.287 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:55:13 smithi183 systemd[1]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-03-26T04:55:14.287 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:55:14 smithi183 systemd[1]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-03-26T04:55:14.287 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:55:14 smithi183 systemd[1]: Stopped Ceph iscsi.iscsi.a for d7d59826-eb26-11ee-95d0-87774f69a715. 2024-03-26T04:55:14.287 INFO:journalctl@ceph.iscsi.iscsi.a.smithi183.stdout:Mar 26 04:55:14 smithi183 systemd[1]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715@iscsi.iscsi.a.service: Consumed 4.303s CPU time. 2024-03-26T04:55:15.783 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:55:15 smithi183 systemd[1]: Stopping Ceph grafana.a for d7d59826-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:55:16.036 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:55:15 smithi183 podman[77624]: 2024-03-26 04:55:15.788563176 +0000 UTC m=+0.094319033 container died 05c46305d2371c8551976e74710de8d17db7ac6f5ac3d3f8f6eae337e91be22c (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, name=ubi8, io.openshift.tags=base rhel8, io.k8s.display-name=Red Hat Universal Base Image 8, io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , architecture=x86_64, com.redhat.component=ubi8-container, vendor=Red Hat, Inc., build-date=2022-04-29T11:48:11.055897, summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, vcs-type=git, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.buildah.version=1.32.0, version=8.5, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664) 2024-03-26T04:55:16.397 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:55:16 smithi183 podman[77624]: 2024-03-26 04:55:16.127675723 +0000 UTC m=+0.433431589 container cleanup 05c46305d2371c8551976e74710de8d17db7ac6f5ac3d3f8f6eae337e91be22c (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a, io.openshift.expose-services=, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-04-29T11:48:11.055897, io.buildah.version=1.32.0, maintainer=Paul Cuzner , io.k8s.display-name=Red Hat Universal Base Image 8, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, version=8.5, com.redhat.component=ubi8-container, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=239.1651231664, vendor=Red Hat, Inc., name=ubi8, vcs-type=git) 2024-03-26T04:55:16.397 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:55:16 smithi183 bash[77624]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a 2024-03-26T04:55:16.397 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:55:16 smithi183 podman[77638]: 2024-03-26 04:55:16.295565707 +0000 UTC m=+0.507581955 container remove 05c46305d2371c8551976e74710de8d17db7ac6f5ac3d3f8f6eae337e91be22c (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-grafana-a, build-date=2022-04-29T11:48:11.055897, distribution-scope=public, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=Red Hat Universal Base Image 8, summary=Grafana Container configured for Ceph mgr/dashboard integration, release=239.1651231664, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, maintainer=Paul Cuzner , vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, io.openshift.tags=base rhel8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.component=ubi8-container, description=Ceph Grafana Container, io.buildah.version=1.32.0, version=8.5, architecture=x86_64, name=ubi8) 2024-03-26T04:55:16.397 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:55:16 smithi183 systemd[1]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-03-26T04:55:16.706 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:55:16 smithi183 systemd[1]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-26T04:55:16.706 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:55:16 smithi183 systemd[1]: Stopped Ceph grafana.a for d7d59826-eb26-11ee-95d0-87774f69a715. 2024-03-26T04:55:16.707 INFO:journalctl@ceph.grafana.a.smithi183.stdout:Mar 26 04:55:16 smithi183 systemd[1]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715@grafana.a.service: Consumed 18.047s CPU time. 2024-03-26T04:55:16.961 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:55:16 smithi183 systemd[1]: Stopping Ceph node-exporter.b for d7d59826-eb26-11ee-95d0-87774f69a715... 2024-03-26T04:55:17.286 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:55:16 smithi183 podman[77753]: 2024-03-26 04:55:16.967403941 +0000 UTC m=+0.119347093 container died 4a3fac17f30bb222a075379ece2658d5215d1d3b44b84bcfb971924fd227c033 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T04:55:17.691 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:55:17 smithi183 podman[77753]: 2024-03-26 04:55:17.298387321 +0000 UTC m=+0.450330478 container cleanup 4a3fac17f30bb222a075379ece2658d5215d1d3b44b84bcfb971924fd227c033 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T04:55:17.691 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:55:17 smithi183 bash[77753]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b 2024-03-26T04:55:17.691 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:55:17 smithi183 podman[77767]: 2024-03-26 04:55:17.482872187 +0000 UTC m=+0.515991809 container remove 4a3fac17f30bb222a075379ece2658d5215d1d3b44b84bcfb971924fd227c033 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-d7d59826-eb26-11ee-95d0-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-26T04:55:17.691 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:55:17 smithi183 systemd[1]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-26T04:55:18.036 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:55:17 smithi183 systemd[1]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-26T04:55:18.036 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:55:17 smithi183 systemd[1]: Stopped Ceph node-exporter.b for d7d59826-eb26-11ee-95d0-87774f69a715. 2024-03-26T04:55:18.036 INFO:journalctl@ceph.node-exporter.b.smithi183.stdout:Mar 26 04:55:17 smithi183 systemd[1]: ceph-d7d59826-eb26-11ee-95d0-87774f69a715@node-exporter.b.service: Consumed 5.032s CPU time. 2024-03-26T04:55:18.332 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T04:55:18.368 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-26T04:55:18.397 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-26T04:55:18.400 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621980/remote/smithi016/crash 2024-03-26T04:55:18.400 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/crash -- . 2024-03-26T04:55:18.444 INFO:teuthology.orchestra.run.smithi016.stderr:tar: /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-26T04:55:18.444 INFO:teuthology.orchestra.run.smithi016.stderr:tar: Error is not recoverable: exiting now 2024-03-26T04:55:18.446 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621980/remote/smithi183/crash 2024-03-26T04:55:18.447 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/crash -- . 2024-03-26T04:55:18.481 INFO:teuthology.orchestra.run.smithi183.stderr:tar: /var/lib/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/crash: Cannot open: No such file or directory 2024-03-26T04:55:18.481 INFO:teuthology.orchestra.run.smithi183.stderr:tar: Error is not recoverable: exiting now 2024-03-26T04:55:18.482 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-26T04:55:18.483 DEBUG:teuthology.orchestra.run.smithi016:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-03-26T04:55:18.518 INFO:teuthology.orchestra.run.smithi016.stdout:2024-03-26T04:20:00.066943+0000 mon.a (mon.0) 869 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 20 pool(s) do not have an application enabled 2024-03-26T04:55:18.519 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-26T04:55:18.519 DEBUG:teuthology.orchestra.run.smithi016:> sudo egrep '\[SEC\]' /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-03-26T04:55:18.590 DEBUG:teuthology.orchestra.run.smithi016:> sudo egrep '\[ERR\]' /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-03-26T04:55:18.667 DEBUG:teuthology.orchestra.run.smithi016:> sudo egrep '\[WRN\]' /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'reached quota' | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(PG_DEGRADED\)' | head -n 1 2024-03-26T04:55:18.738 INFO:teuthology.orchestra.run.smithi016.stdout:2024-03-26T04:20:00.066943+0000 mon.a (mon.0) 869 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 20 pool(s) do not have an application enabled 2024-03-26T04:55:18.739 INFO:tasks.cephadm:Compressing logs... 2024-03-26T04:55:18.739 DEBUG:teuthology.orchestra.run.smithi016:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-26T04:55:18.782 DEBUG:teuthology.orchestra.run.smithi183:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-26T04:55:18.807 INFO:teuthology.orchestra.run.smithi016.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-26T04:55:18.807 INFO:teuthology.orchestra.run.smithi016.stderr:: No such file or directory 2024-03-26T04:55:18.807 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-mon.a.log 2024-03-26T04:55:18.807 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-26T04:55:18.807 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-26T04:55:18.808 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-osd.1.log 2024-03-26T04:55:18.808 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph.audit.log: /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-osd.3.log 2024-03-26T04:55:18.809 INFO:teuthology.orchestra.run.smithi183.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-26T04:55:18.809 INFO:teuthology.orchestra.run.smithi183.stderr:: No such file or directory 2024-03-26T04:55:18.809 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-osd.6.log 2024-03-26T04:55:18.811 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph.audit.log 2024-03-26T04:55:18.811 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-osd.4.log 2024-03-26T04:55:18.811 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-osd.6.log: /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-osd.7.log 2024-03-26T04:55:18.811 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-osd.5.log 2024-03-26T04:55:18.811 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-mgr.y.log 2024-03-26T04:55:18.812 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-osd.0.log 2024-03-26T04:55:18.813 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-osd.4.log: /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-mgr.x.log 2024-03-26T04:55:18.813 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/tcmu-runner.log 2024-03-26T04:55:18.814 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-mgr.y.log: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-26T04:55:18.814 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-26T04:55:18.817 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph.cephadm.log 2024-03-26T04:55:18.817 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/tcmu-runner.log: 85.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-26T04:55:18.817 INFO:teuthology.orchestra.run.smithi183.stderr: 66.9% -- replaced with /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/tcmu-runner.log.gz 2024-03-26T04:55:18.817 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph.log 2024-03-26T04:55:18.818 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-volume.log 81.7% -- replaced with /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-26T04:55:18.818 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-26T04:55:18.818 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-mon.b.log 2024-03-26T04:55:18.822 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-volume.log: /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph.log: 87.8% -- replaced with /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-26T04:55:18.822 INFO:teuthology.orchestra.run.smithi183.stderr: 92.3% -- replaced with /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph.audit.log.gz/var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-mon.b.log: 2024-03-26T04:55:18.823 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph.log 2024-03-26T04:55:18.824 INFO:teuthology.orchestra.run.smithi183.stderr: 92.3% -- replaced with /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-mgr.x.log.gz 2024-03-26T04:55:18.829 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph.cephadm.log: 95.2% -- replaced with /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph.audit.log.gz 2024-03-26T04:55:18.830 INFO:teuthology.orchestra.run.smithi183.stderr: 90.8% -- replaced with /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-26T04:55:18.836 INFO:teuthology.orchestra.run.smithi016.stderr: 89.0% -- replaced with /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph.cephadm.log.gz 2024-03-26T04:55:18.842 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-osd.2.log 2024-03-26T04:55:18.851 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-volume.log 2024-03-26T04:55:18.852 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-osd.2.log: 93.3% -- replaced with /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph.log.gz 2024-03-26T04:55:18.852 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-mon.c.log 2024-03-26T04:55:18.867 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-volume.log: /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-mon.c.log: 90.9% -- replaced with /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-volume.log.gz 2024-03-26T04:55:18.879 INFO:teuthology.orchestra.run.smithi016.stderr: 93.4% -- replaced with /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-03-26T04:55:19.362 INFO:teuthology.orchestra.run.smithi016.stderr: 90.0% -- replaced with /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-mgr.y.log.gz 2024-03-26T04:55:20.290 INFO:teuthology.orchestra.run.smithi016.stderr: 92.4% -- replaced with /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-mon.c.log.gz 2024-03-26T04:55:20.397 INFO:teuthology.orchestra.run.smithi183.stderr: 92.3% -- replaced with /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-mon.b.log.gz 2024-03-26T04:55:21.370 INFO:teuthology.orchestra.run.smithi016.stderr: 91.5% -- replaced with /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-mon.a.log.gz 2024-03-26T04:55:26.893 INFO:teuthology.orchestra.run.smithi183.stderr: 93.8% -- replaced with /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-osd.5.log.gz 2024-03-26T04:55:27.046 INFO:teuthology.orchestra.run.smithi183.stderr: 93.8% -- replaced with /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-osd.6.log.gz 2024-03-26T04:55:27.160 INFO:teuthology.orchestra.run.smithi183.stderr: 93.7% -- replaced with /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-osd.7.log.gz 2024-03-26T04:55:27.586 INFO:teuthology.orchestra.run.smithi016.stderr: 93.8% -- replaced with /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-osd.2.log.gz 2024-03-26T04:55:27.875 INFO:teuthology.orchestra.run.smithi183.stderr: 93.9% -- replaced with /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-osd.4.log.gz 2024-03-26T04:55:27.877 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-26T04:55:27.877 INFO:teuthology.orchestra.run.smithi183.stderr:real 0m9.080s 2024-03-26T04:55:27.877 INFO:teuthology.orchestra.run.smithi183.stderr:user 0m34.127s 2024-03-26T04:55:27.877 INFO:teuthology.orchestra.run.smithi183.stderr:sys 0m1.193s 2024-03-26T04:55:27.908 INFO:teuthology.orchestra.run.smithi016.stderr: 93.9% -- replaced with /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-osd.0.log.gz 2024-03-26T04:55:28.277 INFO:teuthology.orchestra.run.smithi016.stderr: 93.9% -- replaced with /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-osd.1.log.gz 2024-03-26T04:55:28.360 INFO:teuthology.orchestra.run.smithi016.stderr: 93.9% -- replaced with /var/log/ceph/d7d59826-eb26-11ee-95d0-87774f69a715/ceph-osd.3.log.gz 2024-03-26T04:55:28.361 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-03-26T04:55:28.361 INFO:teuthology.orchestra.run.smithi016.stderr:real 0m9.568s 2024-03-26T04:55:28.361 INFO:teuthology.orchestra.run.smithi016.stderr:user 0m40.120s 2024-03-26T04:55:28.361 INFO:teuthology.orchestra.run.smithi016.stderr:sys 0m1.298s 2024-03-26T04:55:28.361 INFO:tasks.cephadm:Archiving logs... 2024-03-26T04:55:28.361 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621980/remote/smithi016/log 2024-03-26T04:55:28.363 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-26T04:55:33.454 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621980/remote/smithi183/log 2024-03-26T04:55:33.455 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-26T04:55:38.329 INFO:tasks.cephadm:Removing cluster... 2024-03-26T04:55:38.329 DEBUG:teuthology.orchestra.run.smithi016:> sudo cephadm rm-cluster --fsid d7d59826-eb26-11ee-95d0-87774f69a715 --force 2024-03-26T04:55:38.529 INFO:teuthology.orchestra.run.smithi016.stdout:Deleting cluster with fsid: d7d59826-eb26-11ee-95d0-87774f69a715 2024-03-26T04:55:38.830 DEBUG:teuthology.orchestra.run.smithi183:> sudo cephadm rm-cluster --fsid d7d59826-eb26-11ee-95d0-87774f69a715 --force 2024-03-26T04:55:39.023 INFO:teuthology.orchestra.run.smithi183.stdout:Deleting cluster with fsid: d7d59826-eb26-11ee-95d0-87774f69a715 2024-03-26T04:55:39.328 INFO:tasks.cephadm:Teardown complete 2024-03-26T04:55:39.329 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-26T04:55:39.340 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-26T04:55:39.340 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-26T04:55:39.342 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-26T04:55:39.411 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-26T04:55:39.420 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-26T04:55:39.420 DEBUG:teuthology.orchestra.run.smithi016:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T04:55:39.423 DEBUG:teuthology.orchestra.run.smithi183:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-26T04:55:39.434 INFO:teuthology.orchestra.run.smithi016.stderr:bash: line 1: ntpq: command not found 2024-03-26T04:55:39.436 INFO:teuthology.orchestra.run.smithi183.stderr:bash: line 1: ntpq: command not found 2024-03-26T04:55:39.439 INFO:teuthology.orchestra.run.smithi016.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T04:55:39.439 INFO:teuthology.orchestra.run.smithi016.stdout:=============================================================================== 2024-03-26T04:55:39.439 INFO:teuthology.orchestra.run.smithi016.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 47 +688us[ +688us] +/- 94ms 2024-03-26T04:55:39.439 INFO:teuthology.orchestra.run.smithi016.stdout:^* hv02.front.sepia.ceph.com 3 7 377 47 -562us[ -600us] +/- 61ms 2024-03-26T04:55:39.439 INFO:teuthology.orchestra.run.smithi016.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 45 +198us[ +198us] +/- 64ms 2024-03-26T04:55:39.439 INFO:teuthology.orchestra.run.smithi016.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:55:39.440 INFO:teuthology.orchestra.run.smithi183.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-26T04:55:39.440 INFO:teuthology.orchestra.run.smithi183.stdout:=============================================================================== 2024-03-26T04:55:39.440 INFO:teuthology.orchestra.run.smithi183.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 104 +675us[ +654us] +/- 94ms 2024-03-26T04:55:39.440 INFO:teuthology.orchestra.run.smithi183.stdout:^* hv02.front.sepia.ceph.com 3 6 377 46 -591us[ -613us] +/- 61ms 2024-03-26T04:55:39.440 INFO:teuthology.orchestra.run.smithi183.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 45 +179us[ +179us] +/- 64ms 2024-03-26T04:55:39.440 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-26T04:55:39.441 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-26T04:55:39.450 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-26T04:55:39.451 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-26T04:55:39.465 DEBUG:teuthology.orchestra.run.smithi016:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-26T04:55:39.557 DEBUG:teuthology.orchestra.run.smithi183:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-26T04:55:39.630 DEBUG:teuthology.orchestra.run.smithi016:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T04:55:39.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:55:39.660 DEBUG:teuthology.orchestra.run.smithi183:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-26T04:55:39.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:55:39.690 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-26T04:55:39.699 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-26T04:55:39.707 INFO:teuthology.task.internal:Duration was 3348.871799 seconds 2024-03-26T04:55:39.708 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-26T04:55:39.716 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-26T04:55:39.716 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-26T04:55:39.718 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-26T04:55:39.758 INFO:teuthology.orchestra.run.smithi016.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T04:55:39.772 INFO:teuthology.orchestra.run.smithi183.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-26T04:55:40.073 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-26T04:55:40.073 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi016.front.sepia.ceph.com 2024-03-26T04:55:40.073 DEBUG:teuthology.orchestra.run.smithi016:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-26T04:55:40.093 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi183.front.sepia.ceph.com 2024-03-26T04:55:40.093 DEBUG:teuthology.orchestra.run.smithi183:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-26T04:55:40.135 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-26T04:55:40.135 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-26T04:55:40.138 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-26T04:55:40.204 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-26T04:55:40.205 DEBUG:teuthology.orchestra.run.smithi016:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-26T04:55:40.207 DEBUG:teuthology.orchestra.run.smithi183:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-26T04:55:40.650 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-26T04:55:40.660 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-26T04:55:40.660 DEBUG:teuthology.orchestra.run.smithi016:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-26T04:55:40.686 DEBUG:teuthology.orchestra.run.smithi183:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-26T04:55:40.714 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-26T04:55:40.723 DEBUG:teuthology.orchestra.run.smithi016:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-26T04:55:40.729 DEBUG:teuthology.orchestra.run.smithi183:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-26T04:55:40.752 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = core 2024-03-26T04:55:40.779 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern = core 2024-03-26T04:55:40.830 DEBUG:teuthology.orchestra.run.smithi016:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-26T04:55:40.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:55:40.844 DEBUG:teuthology.orchestra.run.smithi183:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-26T04:55:40.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-26T04:55:40.884 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-26T04:55:40.894 INFO:teuthology.task.internal:Transferring archived files... 2024-03-26T04:55:40.895 DEBUG:teuthology.misc:Transferring archived files from smithi016:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621980/remote/smithi016 2024-03-26T04:55:40.895 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-26T04:55:40.959 DEBUG:teuthology.misc:Transferring archived files from smithi183:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621980/remote/smithi183 2024-03-26T04:55:40.960 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-26T04:55:40.999 INFO:teuthology.task.internal:Removing archive directory... 2024-03-26T04:55:40.999 DEBUG:teuthology.orchestra.run.smithi016:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-26T04:55:41.002 DEBUG:teuthology.orchestra.run.smithi183:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-26T04:55:41.048 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-26T04:55:41.059 INFO:teuthology.task.internal:Not uploading archives. 2024-03-26T04:55:41.060 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-26T04:55:41.069 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-26T04:55:41.069 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-26T04:55:41.071 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-26T04:55:41.083 INFO:teuthology.orchestra.run.smithi016.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 26 04:55 /home/ubuntu/cephtest 2024-03-26T04:55:41.102 INFO:teuthology.orchestra.run.smithi183.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 26 04:55 /home/ubuntu/cephtest 2024-03-26T04:55:41.103 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-26T04:55:41.113 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-26T04:55:41.132 INFO:teuthology.nuke:Checking targets against current locks 2024-03-26T04:55:41.155 DEBUG:teuthology.nuke:shortname: smithi016 2024-03-26T04:55:41.155 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T04:55:41.179 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi016.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621980', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 03:51:21.281574', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIpwUlkulJHdCP0MC8OTMn5p3Jfm/ZK9sOwgeFKr+GbAa0csykYqsEvQ62XJKhPH3NmE9+u/ddSrluG4s0e/CKk='} 2024-03-26T04:55:41.182 DEBUG:teuthology.nuke:shortname: smithi183 2024-03-26T04:55:41.182 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-26T04:55:41.203 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi183.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-25_21:08:11-orch-squid-distro-default-smithi/7621980', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-26 03:51:21.283180', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBsdLSX8oqYf+V6JXV98ROd9t+2tjF/cUqQj4n2AKMPqehYqqYSAE3gzESTIJSGEATACBb8TkITN6BRY7PvODW8='} 2024-03-26T04:55:41.232 INFO:teuthology.orchestra.console.smithi016:Power off 2024-03-26T04:55:41.232 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-26T04:55:41.258 DEBUG:teuthology.orchestra.console.smithi016:power off output: Chassis Power Control: Down/Off 2024-03-26T04:55:41.258 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T04:55:41.275 INFO:teuthology.orchestra.console.smithi183:Power off 2024-03-26T04:55:41.275 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-26T04:55:41.284 DEBUG:teuthology.orchestra.console.smithi016:check power output: Chassis Power is on 2024-03-26T04:55:41.298 DEBUG:teuthology.orchestra.console.smithi183:power off output: Chassis Power Control: Down/Off 2024-03-26T04:55:41.298 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T04:55:41.321 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is on 2024-03-26T04:55:45.285 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T04:55:45.322 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T04:55:45.395 DEBUG:teuthology.orchestra.console.smithi016:check power output: Chassis Power is on 2024-03-26T04:55:45.433 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is on 2024-03-26T04:55:49.400 DEBUG:teuthology.orchestra.console.smithi016:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T04:55:49.433 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-26T04:55:49.510 DEBUG:teuthology.orchestra.console.smithi016:check power output: Chassis Power is off 2024-03-26T04:55:49.543 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is off 2024-03-26T04:55:49.611 INFO:teuthology.orchestra.console.smithi016:Power off completed 2024-03-26T04:55:49.643 INFO:teuthology.orchestra.console.smithi183:Power off completed 2024-03-26T04:55:49.748 INFO:teuthology.run:Summary data: description: orch/cephadm/with-work/{0-distro/centos_9.stream fixed-2 mode/packaged mon_election/classic msgr/async-v1only start tasks/rados_api_tests} duration: 3348.871798992157 failure_reason: '"2024-03-26T04:20:00.066943+0000 mon.a (mon.0) 869 : cluster 3 [WRN] POOL_APP_NOT_ENABLED: 20 pool(s) do not have an application enabled" in cluster log' flavor: default owner: scheduled_teuthology@teuthology success: false 2024-03-26T04:55:49.748 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-26T04:55:49.831 INFO:teuthology.run:FAIL